Imaging system, method for controlling imaging system, and object recognition system

ABSTRACT

An imaging system of the present disclosure includes: an event detection device that detects an event; and a controller that controls the event detection device. The controller then controls the detection sensitivity of event detection being performed by the event detection device, on the basis of external information. Further, an object recognition system of the present disclosure includes: an event detection device that detects an event; a controller that controls the detection sensitivity of event detection being performed by the event detection device, on the basis of external information; and a recognition processing unit that performs object recognition in an event, on the basis of an event signal output from the event detection device.

TECHNICAL FIELD

The present disclosure relates to an imaging system, a method forcontrolling the imaging system, and an object recognition system.

BACKGROUND ART

Examples of event-driven imaging devices include an asynchronous imagingdevice called a dynamic vision sensor (DVS). An asynchronous imagingdevice can detect an event that a change in the luminance of a pixelthat photoelectrically converts incident light exceeds a predeterminedthreshold. Accordingly, this type of asynchronous imaging device canalso be referred to as an event detection device. Conventionally, anevent detection device has been used as an event-based visual sensorthat is mounted on a vehicle and monitors the road surface on which thevehicle is traveling (see Patent Document 1, for example).

CITATION LIST Patent Document

-   Patent Document 1: Japanese Patent Application Laid-Open No.    2013-79937

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

Meanwhile, in a case where an event detection device is mounted on amobile unit such as a vehicle, appearance of a large number of waterdroplets or the like might be detected as an event in bad weather suchas rain or snow. The event such as the water droplets detected at thistime might then become noise in the object (a vehicle, a pedestrian, orthe like) originally intended to be detected as an event by the eventdetection device, and there is a possibility that this will cause adecrease in the accuracy of object recognition.

The present disclosure aims to provide an imaging system capable ofaccurately recognizing an object in an event without being affected bybad weather such as rain or snow, a method for controlling the imagingsystem, and an object recognition system using the imaging system.

Solutions to Problems

An imaging system of the present disclosure for achieving the aboveobject includes:

an event detection device that detects an event; and

a controller that controls the event detection device.

The controller then controls the detection sensitivity of eventdetection being performed by the event detection device, on the basis ofexternal information.

Further, a method for controlling an imaging system of the presentdisclosure for achieving the above object is a method for controlling animaging system including an event detection device that detects anevent,

the method including

controlling the detection sensitivity of event detection being performedby the event detection device, on the basis of external information.

Further, an object recognition system of the present disclosure forachieving the above object includes:

an event detection device that detects an event;

a controller that controls the detection sensitivity of event detectionbeing performed by the event detection device, on the basis of externalinformation; and

a recognition processing unit that performs object recognition withinthe angle of view of the event detection device, on the basis of anevent signal output from the event detection device.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an example system configuration of animaging system according to a first embodiment of the presentdisclosure.

FIG. 2 is a block diagram showing an example configuration of an eventdetection device in the imaging system according to the firstembodiment.

FIG. 3 is a block diagram showing an example configuration of a pixelarray unit in the event detection device.

FIG. 4 is a circuit diagram showing an example circuit configuration ofa pixel in the event detection device.

FIG. 5 is a block diagram showing an example circuit configuration of anevent detection unit in a pixel in the event detection device.

FIG. 6 is a circuit diagram showing an example configuration of acurrent-voltage conversion unit in the event detection unit.

FIG. 7 is a circuit diagram showing an example configuration of asubtraction unit and a quantization unit in the event detection unit.

FIG. 8 is an exploded perspective view schematically showing a stackedchip structure of the event detection device.

FIG. 9 is a flowchart showing an example of an event detection processaccording to Example 1.

FIG. 10 is an equivalent circuit diagram of a first circuitconfiguration of a current-voltage conversion unit.

FIG. 11 is an equivalent circuit diagram of a second circuitconfiguration of a current-voltage conversion unit.

FIG. 12 is a flowchart showing an example of an event detection processaccording to Example 2.

FIG. 13 is a block diagram showing an example system configuration of anobject recognition system according to the first embodiment.

FIG. 14 is a flowchart showing an example of an event detection processaccording to Example 3.

FIG. 15 is a flowchart showing a specific example in which an objectrecognition process is performed only with the event detection device.

FIG. 16 is a block diagram showing an example system configuration of animaging system according to a second embodiment of the presentdisclosure.

FIG. 17 is a block diagram schematically showing the configuration of aCMOS image sensor that is an example of an imaging device in the imagingsystem according to the second embodiment.

FIG. 18 is a circuit diagram showing an example circuit configuration ofa pixel in the imaging device.

FIG. 19 is a plan view schematically showing a flat chip structure ofthe imaging device.

FIG. 20 is a plan view schematically showing a stacked chip structure ofthe imaging device.

FIG. 21 is a block diagram showing an example system configuration of anobject recognition system according to the second embodiment.

FIG. 22 is a flowchart showing an example of an event detection processaccording to Example 4.

FIG. 23 is a flowchart showing a specific example in which an objectrecognition process is performed with an event detection device and theimaging device.

FIG. 24 is a flowchart showing an example of an event detection processaccording to Example 5.

FIG. 25 is a flowchart showing an example of an event detection processaccording to Example 6.

FIG. 26A is a diagram showing a state in which there is a plurality ofobjects desired to be detected as events, and FIG. 26B is a diagramshowing a state in which many water droplets adhere or do not adhere atall.

FIG. 27 is a flowchart showing an example of an event detection processaccording to Example 7.

FIG. 28 is a block diagram schematically showing an exampleconfiguration of a vehicle control system that is an example of a mobileunit control system to which the technology according to the presentdisclosure can be applied.

FIG. 29 is a diagram showing an example of installation positions ofimaging units in the vehicle control system.

MODES FOR CARRYING OUT THE INVENTION

The following is a detailed description of modes for carrying out thetechnology according to the present disclosure (the modes will behereinafter referred to as “embodiments”), with reference to theaccompanying drawings. The technology according to the presentdisclosure is not limited to the embodiments. In the description below,like components or components having like functions are denoted by likereference numerals, and explanation of them will not be made more thanonce. Note that explanation will be made in the following order.

1. General description of an imaging system and an object recognitionsystem of the present disclosure

2. First embodiment of the present disclosure

2-1. Example configuration of an imaging system according to the firstembodiment

2-2. Example configuration of an event detection device

2-2-1. Example configuration of a pixel array unit

2-2-2. Example configuration of a pixel

2-2-3. Example configuration of an event detection unit

2-2-3-1. Example configuration of a current-voltage conversion unit

2-2-3-2. Example configuration of a subtraction unit and a quantizationunit

2-2-4. Example configuration of a chip structure

2-2-5. Example 1 (an example in which the event detection threshold iscontrolled on the basis of a measurement value of a rain gauge)

2-2-6. Example 2 (an example in which the event detection threshold iscontrolled with the use of a measurement value of a rain gauge and noiseinformation)

2-3. Example configuration of an object recognition system according tothe first embodiment

2-3-1. Example 3 (an example in which an object recognition result isreflected by the event detection threshold)

3. Second embodiment of the present disclosure

3-1. Example configuration of an imaging system according to the secondembodiment

3-2. Example configuration of an imaging device

3-2-1. Example configuration of a CMOS image sensor

3-2-2. Example configuration of a pixel

3-2-3. Example configuration of a chip structure

3-2-3-1. Flat chip structure (a so-called flat structure)

3-2-3-2. Stacked chip structure (a so-called stack structure)

3-3. Example configuration of an object recognition system according tothe second embodiment

3-3-1. Example 4 (an example in which an object recognition result isreflected by the event detection threshold)

3-3-2. Example 5 (an example in which the event detection threshold iscontrolled with the use of a measurement value of a rain gauge and noiseinformation)

3-3-3. Example 6 (an example in which an object recognition process isperformed twice before and after control of the event detectionthreshold)

3-3-4. Example 7 (an example in which the event detection threshold iscontrolled region by region)

4. Modifications

5. Example applications of the technology according to the presentdisclosure

5-1. Example applications to mobile units

6. Configurations embodying the present disclosure

General Description of an Imaging System and an Object RecognitionSystem of the Present Disclosure

In an imaging system and an object recognition system of the presentdisclosure, an event detection device may have a configuration thatincludes an event detection unit that detects, as an event, that achange in the luminance of a pixel that photoelectrically convertsincident light exceeds a detection threshold. Further, the imagingsystem of the present disclosure is preferably mounted on a mobile unit.

Furthermore, in the imaging system and the object recognition system ofthe present disclosure including the preferable configuration describedabove, the controller can be designed to control the detection thresholdfor the event detection unit on the basis of external information.Specifically, when the external information is information indicatingbad weather, the controller can perform control to raise the detectionthreshold for the event detection unit. When receiving externalinformation indicating that the weather has recovered after raising thedetection threshold for the event detection unit, the controller canalso perform control to lower the detection threshold for the eventdetection unit or return the detection threshold for the event detectionunit to the initial setting value.

Further, in the imaging system and the object recognition system of thepresent disclosure including the preferable configuration describedabove, the event detection unit may have a configuration including acurrent-voltage conversion unit that converts the photocurrent of apixel into a voltage corresponding to the photocurrent, and thecurrent-voltage conversion unit can switch between a circuitconfiguration in which transistors are cascade-connected and a circuitconfiguration in which the transistors are not cascade-connected. Inthis case, the controller can control the detection threshold for theevent detection unit, by switching the circuit configuration of thecurrent-voltage conversion unit on the basis of external information.

Also, in the imaging system and the object recognition system of thepresent disclosure including the preferable configuration describedabove, the event detection unit may have a configuration including asubtraction unit that includes a first capacitive element and a secondcapacitive element, and calculates a difference signal of a voltagebetween different timings, the voltage corresponding to the photocurrentof a pixel. In this case, the controller can control the detectionthreshold for the event detection unit, by changing the capacitanceratio between the first capacitive element and the second capacitiveelement of the subtraction unit, on the basis of external information.

Also, in the imaging system and the object recognition system of thepresent disclosure including the preferable configuration describedabove, the event detection unit may have a configuration including aquantization unit that quantizes the difference signal from thesubtraction unit into a digital signal, by comparing the differencesignal with a threshold voltage. In this case, the controller cancontrol the detection threshold for the event detection unit, byadjusting the threshold voltage of the quantization unit on the basis ofexternal information.

Further, in the imaging system and the object recognition system of thepresent disclosure including the preferable configuration describedabove, the controller can control the detection sensitivity of eventdetection, on the basis of external information and the number of eventsdetected by the event detection device.

Also, in the object recognition system of the present disclosure, thecontroller can perform control to raise the detection threshold for theevent detection unit to a value within the range in which an objectwithin the angle of view of the event detection device can berecognized, and a recognition processing unit can recognize the objectwhen the area that can be detected as the object within the angle ofview is equal to or larger than a predetermined threshold.

Further, the object recognition system of the present disclosureincluding the preferable configuration described above can include asynchronous imaging device that performs imaging at a fixed frame rate.

Further, in the object recognition system of the present disclosureincluding the preferable configuration described above, the recognitionprocessing unit can detect an object using information from the imagingdevice when the event detection device cannot detect the object, andrecognize the object when the area that can be detected as the objectwithin the angle of view is equal to or larger than the predeterminedthreshold. Also, the recognition processing unit can determine that therecognition has been successfully completed when the region of theobject with respect to the angle of view is equal to or larger than acertain proportion, and determine that the recognition has failed whenthe region of the object with respect to the angle of view is smallerthan the certain proportion.

Further, in the object recognition system of the present disclosureincluding the preferable configuration described above, the controllercan perform control to raise the detection threshold for the eventdetection unit, when the external information is information indicatingbad weather, and the number of events detected by the event detectiondevice is equal to or larger than a predetermined threshold. Further,after the controller raises the detection threshold for the eventdetection unit, when a result of recognition performed by therecognition processing unit indicates a success, and the controllerreceives external information indicating that the weather has recovered,the controller can perform control to lower the detection threshold forthe event detection unit or return the detection threshold for the eventdetection unit to the initial setting value.

Also, in the object recognition system of the present disclosureincluding the preferable configuration described above, when theexternal information is information indicating bad weather, and a resultof recognition performed by the recognition processing unit indicates afailure, the controller can perform control to raise the detectionthreshold for the event detection unit. Further, after the controllerraises the detection threshold for the event detection unit, when aresult of recognition performed by the recognition processing unitindicates a success, and the controller receives external informationindicating that the weather has recovered, the controller can performcontrol to lower the detection threshold for the event detection unit orreturn the detection threshold for the event detection unit to theinitial setting value.

First Embodiment of the Present Disclosure Example Configuration of anImaging System According to the First Embodiment

FIG. 1 is a block diagram showing an example system configuration of animaging system according to a first embodiment of the presentdisclosure.

As shown in FIG. 1, an imaging system 1A according to the firstembodiment includes an event detection device 10, a controller 30, adata processing unit 40, and an image recording unit 50. The imagingsystem 1A according to the first embodiment can be mounted on a mobileunit such as a vehicle, and be used.

In a case where the imaging system 1A is mounted on a vehicle and isused, for example, the imaging system 1A is disposed and is used at oneor more predetermined positions of the vehicle, such as the front endedge of the vehicle, a side mirror, the rear bumper, a rear door, or anupper portion of the front windshield inside the vehicle. Exampleapplications of the technology according to the present disclosure(which is the imaging system 1A according to the first embodiment) willbe described later in detail.

As the event detection device 10, an asynchronous imaging device calledDVS that detects an event that a change in the luminance of a pixel thatphotoelectrically converts incident light exceeds a predetermineddetection threshold can be used. An asynchronous imaging device is animaging device that detects an event asynchronously with a verticalsynchronization signal, as opposed to a synchronous imaging device thatperforms imaging in synchronization with a vertical synchronizationsignal. The event detection device 10 formed with an asynchronousimaging device will be described later in detail.

The controller 30 is formed with a processor (CPU), for example, andcontrols detection sensitivity of event detection being performed by theevent detection device 10, on the basis of external information suppliedfrom the outside. Examples of the external information supplied from theoutside include information about an environment sensor for acquiringinformation regarding the present weather such as rain, snow, and hail.

The environment sensor may be formed with at least one of the followingsensors: a rain gauge that measures an amount of rain, a raindrop sensorthat detects rain, a fog sensor that detects a fog, a solar radiationsensor that detects a degree of solar radiation, or a snow sensor thatdetects a snowfall, for example. The environment sensor is disposed atone or more of the following positions: the front end edge of thevehicle, a side mirror, the rear bumper, a rear door, or an upperportion of the front windshield inside the vehicle, for example.

Here, the information about the environment sensor described above hasbeen described as an example of the external information supplied to thecontroller 30, but the information is not necessarily the informationabout the environment sensor. In addition to the information about theenvironment sensor, the external information may include informationregarding presence/absence of an operation of the windshield wipers,weather information that can be acquired through the Internet or aglobal positioning system (GPS), or the like.

Under the control of the controller 30, the data processing unit 40performs predetermined data processing on an event signal (event data)that is output from the event detection device 10 and indicates theoccurrence of an event. Through this data processing unit 40, thecontroller 30 can recognize that the event detection device 10 hasdetected an event. The image recording unit 50 records image dataprocessed by the data processing unit 40.

Example Configuration of the Event Detection Device

FIG. 2 is a block diagram showing an example configuration of the eventdetection device 10 in the imaging system 1A according to the firstembodiment having the configuration described above.

As shown in FIG. 2, the event detection device 10 includes a pixel arrayunit 12 in which a plurality of pixels 11 is two-dimensionally arrangedin a matrix (array). Each of the plurality of pixels 11 generates, as apixel signal, an analog signal of a voltage corresponding to aphotocurrent as an electric signal generated by photoelectricconversion. Further, each of the plurality of pixels 11 detects thepresence/absence of an event, depending on whether or not a changeexceeding a predetermined detection threshold has occurred in thephotocurrent corresponding to the luminance of incident light. In otherwords, each of the plurality of pixels 11 detects an event that a changein luminance exceeds a predetermined threshold.

In addition to the plurality of pixel array unit 12, the event detectiondevice 10 includes a drive unit 13, an arbiter unit (arbitration unit)14, a column processing unit 15, and a signal processing unit 16, as aperipheral circuit unit of the pixel array unit 12.

When having detected an event, each of the plurality of pixels 11outputs a request for requesting an output of event data indicating theoccurrence of the event, to the arbiter unit 14. In a case where each ofthe plurality of pixels 11 has received, from the arbiter unit 14, aresponse indicating permission for an output of event data, the eventdata is then output to the drive unit 13 and the signal processing unit16. Further, a pixel 11 that has detected an event outputs an analogpixel signal generated by photoelectric conversion, to the columnprocessing unit 15.

The drive unit 13 drives each pixel 11 of the pixel array unit 12. Forexample, the drive unit 13 detects an event, drives the pixel 11 thathas output the event data, and outputs an analog pixel signal of thepixel 11 to the column processing unit 15.

The arbiter unit 14 arbitrates a request for requesting an output ofevent data supplied from each of the plurality of pixels 11, andtransmits, to the pixels 11, a response based on a result of thearbitration (permission/non-permission of an output of event data) and areset signal for resetting event detection.

The column processing unit 15 includes an analog-digital conversion unitformed with a set of analog-digital converters provided for therespective pixel columns of the pixel array unit 12, for example. Theanalog-digital converters can be single-slope analog-digital converters,for example.

For each pixel column of the pixel array unit 12, the column processingunit 15 performs a process of converting analog pixel signals outputfrom the pixels 11 in the column into digital signals. The columnprocessing unit 15 can also perform a correlated double sampling (CDS)process on digitized pixel signals.

The signal processing unit 16 performs predetermined signal processingon digitized pixel signals supplied from the column processing unit 15and the event data output from the pixel array unit 12, and outputs theevent data and the pixel signals subjected to the signal processing.

As described above, a change in a photocurrent generated in a pixel 11can also be regarded as a change (luminance change) in the amount oflight entering the pixel 11. Accordingly, an event can also be regardedas a light amount change (luminance change) in the pixel 11 exceeding apredetermined threshold. Event data indicating the occurrence of anevent includes at least location information such as the coordinatesindicating the position of the pixel 11 in which a light amount changeas an event has occurred. The event data can also include the polarityof the light amount change, in addition to the location information.

As for a series of event data output at the timing when an event hasoccurred at a pixel 11, the event data can implicitly include timeinformation indicating the relative time at which the event occurred, aslong as the intervals between the pieces of the event data aremaintained as they were at the time of the event occurrence. However,when the intervals between the pieces of event data are no longermaintained as they were at the time of the event occurrence due to thestorage of the event data in a memory or the like, the time informationimplicitly included in the event data will be lost. Therefore, thesignal processing unit 16 incorporates the time information indicatingthe relative time at which the event occurred, such as a time stamp,into the event data, before the intervals between the pieces of theevent data are no longer maintained as they were at the time of theevent occurrence.

Example Configuration of the Pixel Array Unit

FIG. 3 is a block diagram showing an example configuration of the pixelarray unit 12 in the event detection device 10.

In the pixel array unit 12 in which the plurality of pixels 11 istwo-dimensionally arranged in a matrix, each of the plurality of pixels11 includes a light receiving unit 61, a pixel signal generation unit62, and an event detection unit 63.

In a pixel 11 having the above configuration, the light receiving unit61 photoelectrically converts incident light, to generate aphotocurrent. Under the control of the drive unit 13 (see FIG. 2), thelight receiving unit 61 then supplies a signal of the voltagecorresponding to the photocurrent generated by photoelectricallyconverting the incident light, to either the pixel signal generationunit 32 or the event detection unit 63.

The pixel signal generation unit 62 generates the signal of the voltagecorresponding to the photocurrent supplied from the light receiving unit61, as an analog pixel signal SIG. The pixel signal generation unit 62then supplies the generated analog pixel signal SIG to the columnprocessing unit 15 (see FIG. 2) via a vertical signal line VSL providedfor each pixel column of the pixel array unit 12.

The event detection unit 63 detects presence/absence of occurrence of anevent, depending on whether or not the amount of change in thephotocurrent from each corresponding light receiving unit 61 exceeds apredetermined detection threshold. The event includes an on-eventindicating that the amount of change in the photocurrent exceeds theupper limit threshold, and an off-event indicating that the amount ofchange falls below the lower limit threshold, for example. Further, theevent data indicating the occurrence of the event includes one bitindicating a result of on-event detection, and one bit indicating aresult of off-event detection, for example. Note that the eventdetection unit 63 can be designed to detect only an on-event.

When an event has occurred, the event detection unit 63 outputs arequest for requesting an output of event data indicating the occurrenceof the event, to the arbiter unit 14 (see FIG. 2). In a case where aresponse to the request is received from the arbiter unit 14, the eventdetection unit 63 then outputs the event data to the drive unit 13 andthe signal processing unit 16.

Example Circuit Configuration of a Pixel

FIG. 4 is a circuit diagram showing an example circuit configuration ofa pixel 11 of the pixel array unit 12 in the event detection device 10.

As described above, each of the plurality of pixels 11 includes thelight receiving unit 61, the pixel signal generation unit 62, and theevent detection unit 63.

In the pixel 11 having the above configuration, the light receiving unit61 includes a light receiving element (a photoelectric conversionelement) 611, a transfer transistor 612, and a transfer transistor 613.The transfer transistor 612 and the transfer transistor 613 can beN-type metal oxide semiconductor (MOS) transistors, for example. Thetransfer transistor 612 and the transfer transistor 613 are connected inseries to each other.

The light receiving element 611 is connected between the ground and acommon connection node N₁ of the transfer transistor 612 and thetransfer transistor 613, and photoelectrically converts incident lightto generate electric charge of the amount corresponding to the amount ofthe incident light.

A transfer signal TRG is supplied from the drive unit 13 shown in FIG. 2to the gate electrode of the transfer transistor 612. The transfertransistor 612 enters an on-state in response to the transfer signalTRG, to supply the pixel signal generation unit 62 with an electricsignal generated by the light receiving element 611 performingphotoelectric conversion.

A control signal OFG is supplied from the drive unit 13 to the gateelectrode of the transfer transistor 613. The transfer transistor 613enters an on-state in response to the control signal OFG, to supply theevent detection unit 63 with an electric signal generated by the lightreceiving element 611 performing photoelectric conversion. The electricsignal supplied to the event detection unit 63 is a photocurrent formedwith electric charge.

The pixel signal generation unit 62 includes a reset transistor 621, anamplification transistor 622, a selection transistor 623, and a floatingdiffusion layer 624. The reset transistor 621, the amplificationtransistor 622, and the selection transistor 623 can be N-type MOStransistors, for example.

The electric charge photoelectrically converted by the light receivingelement 611 of the light receiving unit 61 is supplied to the pixelsignal generation unit 62 by the transfer transistor 612. The electriccharge supplied from the light receiving unit 61 is accumulated in thefloating diffusion layer 624. The floating diffusion layer 624 generatesa voltage signal of the voltage value corresponding to the amount of theaccumulated electric charge. That is, the floating diffusion layer 624is a charge-voltage conversion unit that converts electric charge into avoltage.

The reset transistor 621 is connected between the power supply line of apower supply voltage V_(DD) and the floating diffusion layer 624. Areset signal RST is supplied from the drive unit 13 to the gateelectrode of the reset transistor 621. The reset transistor 621 entersan on-state in response to the reset signal RST, to initialize (reset)the floating diffusion layer 624.

The amplification transistor 622 is connected in series to the selectiontransistor 623 between the power supply line of the power supply voltageV_(DD) and the vertical signal line VSL. The amplification transistor622 amplifies the voltage signal subjected to charge-voltage conversionby the floating diffusion layer 624.

A selection signal SEL is supplied from the drive unit 13 to the gateelectrode of the selection transistor 623. The selection transistor 623enters an on-state in response to the selection signal SEL, to outputthe voltage signal amplified by the amplification transistor 622 as ananalog pixel signal SIG to the column processing unit 15 (see FIG. 2)via the vertical signal line VSL.

In the event detection device 10 including the pixel array unit 12 inwhich the pixels 11 having the above configuration are two-dimensionallyarranged, the drive unit 13 is instructed to start event detection bythe controller 30 shown in FIG. 1. Instructed to start event detection,the drive unit 13 then supplies the control signal OFG to the transfertransistor 613 of the light receiving unit 61, to drive the transfertransistor 613 to supply the photocurrent corresponding to the electriccharge generated in the light receiving element 611 to the eventdetection unit 63.

When an event is detected in a pixel 11, the drive unit 13 then turnsoff the transfer transistor 613 of the pixel 11, to stop the supply ofphotocurrent to the event detection unit 63. Next, the drive unit 13supplies the transfer signal TRG to the transfer transistor 612, todrive the transfer transistor 612 to transfer the electric chargephotoelectrically converted by the light receiving element 611 to thefloating diffusion layer 624.

In this manner, the event detection device 10 including the pixel arrayunit 12 in which the pixels 11 having the above configuration aretwo-dimensionally arranged outputs only a pixel signal of the pixel 11from which an event has been detected, to the column processing unit 15.Thus, power consumption by the event detection device 10 and imageprocessing load can be made lower than those in a case where pixelsignals of all the pixels are output, regardless of the presence/absenceof an event.

Note that the configuration of a pixel 11 described herein is anexample, and is not limited to this example configuration. For example,in a case where there is no need to output a pixel signal, a pixelconfiguration not including the pixel signal generation unit 62 can beadopted. In the case of this pixel configuration, the transfertransistor 612 is only required to be removed from the light receivingunit 61. Further, the column processing unit 15 in FIG. 2 can bedesigned not to have an analog-digital conversion function. With a pixelconfiguration that does not output any pixel signal, the size of theevent detection device 10 can be reduced.

Example Configuration of the Event Detection Unit

FIG. 5 is a block diagram showing an example circuit configuration ofthe event detection unit 63 in a pixel 11 of the event detection device10.

As shown in FIG. 5, the event detection unit 63 according to thisexample includes a current-voltage conversion unit 631, a buffer 632, asubtraction unit 633, a quantization unit 634, and a transfer unit 635.

The current-voltage conversion unit 631 converts a photocurrent suppliedfrom the light receiving unit 63 of the pixel 11 into a voltage signalof the logarithm of the photocurrent (this voltage signal will behereinafter referred to as “optical voltage” in some cases), andsupplies the voltage signal to the buffer 632. The buffer 632 buffersthe optical voltage supplied from the current-voltage conversion unit631, and supplies the optical voltage to the subtraction unit 633.

The subtraction unit 633 calculates a difference between the opticalvoltage at the current time and an optical voltage at a time slightlydifferent from the current time, and supplies a difference signalcorresponding to the difference to the quantization unit 634. Thequantization unit 634 quantizes the difference signal supplied from thesubtraction unit 633 into a digital signal, and supplies the digitalvalue of the difference signal to the transfer unit 635.

When the digital value of the difference signal is supplied from thequantization unit 634, the transfer unit 635 supplies a request forrequesting transmission of event data to the arbiter unit 14. Receivinga response to the request, or a response indicating that an output ofevent data is allowed, from the arbiter unit 14, the transfer unit 635then supplies event data to the drive unit 13 and the signal processingunit 16 in accordance with the digital value of the difference signalsupplied from the quantization unit 634.

Next, example configurations of the current-voltage conversion unit 631,the subtraction unit 633, and the quantization unit 634 in the eventdetection unit 63 are described.

Example Configuration of the Current-Voltage Conversion Unit

FIG. 6 is a circuit diagram showing an example configuration of thecurrent-voltage conversion unit 631 in the event detection unit 63.

As shown in FIG. 6, the current-voltage conversion unit 631 according tothis example has a circuit configuration that includes a transistor6311, a transistor 6312, a transistor 6313, a transistor 6314, and atransistor 6315, as well as a switch element SW₁, a switch element SW₂,a switch element SW₃, and a switch element SW₄. N-type MOS transistorscan be used as the transistor 6311, the transistor 6313, the transistor6314, and the transistor 6315, and a P-type MOS transistor can be usedas the transistor 6312.

The transistor 6311 and the transistor 6314 are connected in seriesbetween the power supply line of the power supply voltage V_(DD) and asignal input line L. The transistor 6312, the transistor 6315, and thetransistor 6313 are connected in series between the power supply line ofthe power supply voltage V_(DD) and the ground. Further, the gateelectrode of the transistor 6311 and the input terminal of the buffer632 shown in FIG. 5 are connected to a common connection node N₂ of thetransistor 6312 and the transistor 6315.

A predetermined bias voltage V_(bias) is applied to the gate electrodeof the transistor 6312. As a result, the transistor 6312 supplies aconstant current to the transistor 6313. A photocurrent is input fromthe light receiving unit 61 to the gate electrode of the transistor 6313through the signal input line L. The drain electrode of the transistor6311 is connected to the power supply line of the power supply voltageV_(DD), and forms a source follower configuration.

The switch element SW₁ is connected between the signal input line L anda common connection node N₃ of the transistor 6311 and the transistor6314. That is, the switch element SW₁ is connected in parallel to thetransistor 6314. The switch element SW₂ is connected between the commonconnection node N₃ of the transistor 6311 and the transistor 6314, andthe gate electrode of the transistor 6315.

The switch element SW₃ is connected between the common connection nodeN₂ of the transistor 6312 and the transistor 6315, and a commonconnection node N₄ of the transistor 6315 and the transistor 6313. Thatis, the switch element SW₃ is connected in parallel to the transistor6315. The switch element SW₄ is connected between the gate electrode ofthe transistor 6314 and the common connection node N₄ of the transistor6315 and the transistor 6313.

The gate electrode of the transistor 6313 is connected to the sourceelectrode of the transistor 6311 having a source follower configuration,via the transistor 6314. A photocurrent from the light receiving unit 61is then converted into an optical voltage corresponding to the logarithmof the photocurrent by the transistor 6311 having a source followerconfiguration and the transistor 6313.

The current-voltage conversion unit 631 having the above configurationcan switch between a circuit configuration in which transistors arecascade-connected and a circuit configuration in which transistors arenot cascade-connected, by controlling switching on (closing) and off(opening) of the switch element SW₁, the switch element SW₂, the switchelement SW₃, and the switch element SW₄. By switching circuitconfigurations in this manner, the current-voltage conversion unit 631can then control the detection threshold of the event detection unit 63,which is the detection sensitivity of event detection being performed bythe event detection device 10. The circuit configuration switching to beperformed by the current-voltage conversion unit 631 will be describedlater in detail.

Example Configuration of the Subtraction Unit and the Quantization Unit

FIG. 7 is a circuit diagram showing an example configuration of thesubtraction unit 633 and the quantization unit 634 in the eventdetection unit 63.

Example Configuration of the Subtraction Unit

The subtraction unit 633 according to this example includes a capacitiveelement 6331 as a first capacitive element, an operational amplifier6332, a capacitive element 6333 as a second capacitive element, and aswitch element 6334.

One end of the capacitive element 6331 is connected to the outputterminal of the buffer 632 shown in FIG. 5, and the other end of thecapacitive element 6331 is connected to the input terminal of theoperational amplifier 6332. With this arrangement, an optical voltagesupplied from the buffer 632 is input to the input terminal of theoperational amplifier 6332 via the capacitive element 6331.

The capacitive element 6333 is connected in parallel to the operationalamplifier 6332. The switch element 6334 is connected between both endsof the capacitive element 6333. To the switch element 6334, a resetsignal is supplied as a control signal for opening or closing the switchelement 6334, from the arbiter unit 14 shown in FIG. 2. In accordancewith the reset signal, the switch element 6334 opens or closes the pathconnecting both ends of the capacitive element 6333.

In the subtraction unit 633 having the above configuration, the opticalvoltage that is input to the terminal of the capacitive element 6331 onthe side of the buffer 632 when the switch element 6334 is turned on(closed) is represented by V_(init). When the optical voltage V_(init)is input to the terminal of the capacitive element 6331 on the side ofthe buffer 632, the terminal on the opposite side becomes a virtualground terminal. The potential of this virtual ground terminal is set tozero, for convenience. At this point of time, the electric chargeQ_(init) accumulated in the capacitive element 6331 is expressed byEquation (1) shown below, where C₁ represents the capacitance value ofthe capacitive element 6331.

Q _(init) =C ₁ ×V _(init)  (1)

Further, in a case where the switch element 6334 is in an on-state, bothends of the capacitive element 6333 are short-circuited, andaccordingly, the electric charge accumulated in the capacitive element6333 is zero. After that, the switch element 6334 is turned off(opened). The optical voltage of the terminal of the capacitive element6331 on the side of the buffer 632 in a case where the switch element6334 is in an off-state is represented as V_(after). The electric chargeQ_(after) accumulated in the capacitive element 6331 in a case where theswitch element 6334 enters an off-state is expressed by Equation (2)shown below.

Q _(after) =C ₁ ×V _(after)  (2)

Where the capacitance value of the capacitive element 6333 isrepresented by C₂, and the output voltage of the operational amplifier6332 is represented by V_(out), the electric charge Q₂ accumulated inthe capacitive element 6333 is expressed by Equation (3) shown below.

Q ₂ =−C ₂ ×V _(out)  (3)

Before and after the switch element 6334 is turned off, the total chargeamount obtained by combining the charge amount of the capacitive element6331 and the charge amount of the capacitive element 6333 does notchange. Accordingly, Equation (4) shown below is established.

Q _(init) =Q _(after) +Q ₂  (4)

Where Equations (1) to (3) are substituted into Equation (4), Equation(5) shown below is obtained.

V _(out)=−(C ₁ /C ₂)×(V _(after) −V _(init))  (5)

According to Equation (5), the subtraction unit 633 performs subtractionbetween the optical voltage V_(init) and the optical voltage V_(after),which is calculation of a difference signal V_(out) corresponding to thedifference (V_(init)−V_(after)) between the optical voltage V_(init) andthe optical voltage V_(after). Also, according to Equation (5), thesubtraction gain of the subtraction unit 633 is C₁/C₂. Since it isnormally desired to maximize the subtraction gain of the subtractionunit 633, it is preferable to design the capacitance value C₁ of thecapacitive element 6331 to be large, and the capacitance value C₂ of thecapacitive element 6333 to be small.

When the capacitance value C₂ of the capacitive element 6333 is toosmall, on the other hand, kTC noise increases, and noise characteristicsmight deteriorate. Therefore, the reduction of the capacitance value C₂of the capacitive element 6333 is limited within a range in which noisecan be tolerated. Further, since the event detection unit 63 includingthe subtraction unit 633 is mounted in each pixel 11, constraints areimposed on the areas of the capacitive element 6331 and the capacitiveelement 6333. With these aspects taken into account, the capacitancevalue C₁ of the capacitive element 6331 and the capacitance value C₂ ofthe capacitive element 6333 are determined.

In the subtraction unit 633 having the above configuration, variablecapacitive elements each having a variable capacitance value can be usedas the capacitive element 6331 that is the first capacitive element, andthe capacitive element 6333 that is the second capacitive element.Further, the capacitance ratio (C₁/C₂) between the capacitive element6331 and the capacitive element 6333 is changed, so that the detectionthreshold of the event detection unit 63, which is the detectionsensitivity of the event detection being performed by the eventdetection device 10, can be controlled.

Example Configuration of the Quantization Unit

In FIG. 7, the quantization unit 634 includes a comparator 6341. Thecomparator 6341 receives a difference signal from the subtraction unit430 (which is an output signal of the operational amplifier 6332) as anon-inverting (+) input, and receives a predetermined threshold voltageV_(th) as an inverting (−) input. The comparator 6341 then compares thedifference signal V_(out) from the subtraction unit 430 with thepredetermined threshold voltage V_(th), and outputs a high level or alow level representing the comparison result as the quantization valueof the difference signal V_(out) to the transfer unit 635 shown in FIG.5.

In the quantization unit 634 having the above configuration, thethreshold voltage V_(th) can be variable. Further, the threshold voltageV_(th) for the quantization unit 634 is adjusted, so that the detectionthreshold for the event detection unit 63, which is the detectionsensitivity of the event detection being performed by the eventdetection device 10, can be controlled.

In a case where the occurrence of a change in the light amount (a changein luminance) as an event is recognized from the quantization value ofthe difference signal V_(out) from the quantization unit 634, or wherethe difference signal V_(out) is greater (or smaller) than thepredetermined threshold voltage V_(th), the transfer unit 635 outputshigh-level event data indicating the occurrence of an event to thesignal processing unit 16 in FIG. 2, for example.

In FIG. 2, the signal processing unit 16 outputs the event data suppliedfrom the transfer unit 635, including location information about thepixel 11 that has detected the event indicated by the event data, timeinformation indicating the time when the event has occurred, andfurther, if necessary, polarity information about the light amountchange as the event.

The data format of the event data including the location informationabout the pixel 11 that has detected the event, the time informationindicating the time when the event has occurred, and the polarityinformation about the light amount change as the event can be a dataformat called Address Event Representation (AER), for example.

Note that an optical filter such as a color filter that transmitspredetermined light is provided in the pixel 11, so that the pixel 11can receive desired light as incident light. For example, in a casewhere the pixel 11 receives visible light as incident light, the eventdata indicates the occurrence of a change in a pixel value in an imageshowing a visible object. Also, in a case where the pixel 11 is toreceive infrared rays, millimeter waves, or the like for distancemeasurement as incident light, for example, the event data indicates theoccurrence of a change in the distance to the object. Further, in a casewhere the pixel 11 is to receive infrared rays for measuring temperatureas incident light, for example, the event data indicates the occurrenceof a change in the temperature of the object. In this embodiment, thepixel 11 is to receive visible light as incident light.

Example Configuration of a Chip Structure

The chip (semiconductor integrated circuit) structure of the eventdetection device 10 described above can be a stacked chip structure, forexample. FIG. 8 is an exploded perspective view schematically showing astacked chip structure of the event detection device 10.

As shown in FIG. 8, the stacked chip structure, which is a stackstructure, is a structure in which at least two chips that is a lightreceiving chip 101 as a first chip and a detection chip 102 as a secondchip are stacked. Further, in the circuit configuration of the pixels 11shown in FIG. 4, each light receiving element 611 is disposed on thelight receiving chip 101, and all the elements other than the lightreceiving elements 611, the elements of the other circuit portions ofthe pixels 11, and the like are disposed on the detection chip 102. Thelight receiving chip 101 and the detection chip 102 are electricallyconnected via connecting portions such as vias, Cu—Cu joints, or bumps.

Note that, although an example configuration in which the lightreceiving elements 611 are disposed on the light receiving chip 101, andthe elements other than the light receiving elements 611, the elementsof the other circuit portions of the pixels 11, and the like aredisposed on the detection chip 102 has been described herein, thisembodiment is not limited to this example configuration.

For example, in the circuit configuration of the pixels 11 shown in FIG.4, each element of the light receiving units 61 may be disposed on thelight receiving chip 101, and the elements other than the lightreceiving units 61, the elements of the other circuit portions of thepixels 11, and the like may be disposed on the detection chip 102.Alternatively, each element of the light receiving units 61, and thereset transistors 621 and the floating diffusion layers 624 of the pixelsignal generation units 62 may be disposed on the light receiving chip101, and the other elements may be disposed on the detection chip 102.Further, some of the elements constituting the event detection units 63,together with the respective elements of the light receiving units 61and the like, may be disposed on the light receiving chip 101.

In the description below, a specific example of an event detectionprocess to be performed in the imaging system 1A according to the firstembodiment having the above configuration will be explained. The eventdetection described below is basically performed under the control ofthe controller 30 of the imaging system 1. This aspect applies to eachof the examples described later.

Example 1

Example 1 is an example of the imaging system 1A according to the firstembodiment in which a rain gauge that measures an amount of rain is usedas an environment sensor that provides external information, and thedetection threshold for the event detection units 63 (this thresholdwill be hereinafter simply described as an “event detection threshold”in some cases) is controlled on the basis of a measurement value of therain gauge. An example flow in an event detection process according toExample 1 is shown in a flowchart in FIG. 9. The event detection processaccording to Example 1 is an example process according to a controlmethod of the present disclosure, by which the detection sensitivity ofevent detection being performed by the event detection device iscontrolled on the basis of external information.

The controller 30 sets a predetermined initial setting value as thedetection threshold for the event detection units 63, and causes theevent detection device 10 to perform imaging (step S11). The eventdetection threshold can be set depending on the circuit configuration ofthe current-voltage conversion unit 631 described above, the capacitanceratio between the capacitive element 6331 and the capacitive element6333 in the subtraction unit 633, or the threshold voltage V_(th) in thequantization unit 634. This aspect applies to the examples describedlater.

While imaging is being performed by the event detection device 10, thecontroller 30 acquires a measurement value of the rain gauge provided asexternal information (step S12), and then determines whether or not themeasurement value of the rain gauge is equal to or greater than apredetermined threshold (step S13). If the measurement value is neitherequal to nor greater than the predetermined threshold (NO in S13), theprocess returns to step S11, and the imaging by the event detectiondevice 10 continues.

If the measurement value of the rain gauge is equal to or greater thanthe predetermined threshold (YES in S13), the controller 30 performscontrol to raise the detection threshold for the event detection units63 by a certain value, for example (step S14). Raising the detectionthreshold for the event detection units 63 means lowering the detectionsensitivity of the event detection being performed by the eventdetection device 10 (the gain of the event detection device 10). Aspecific example of the event detection threshold control will bedescribed later.

The measurement value of the rain gauge being equal to or greater thanthe predetermined threshold means that the amount of rain is large. In asituation where the amount of rain is large, there might be a case wherethe event detection device 10 detects generation of a large number ofwater droplets (raindrops) or the like as an event. In this case, thereis a possibility that the water droplets (raindrops) or the likedetected as an event will become noise for the object (a vehicle, apedestrian, or the like) originally desired to be detected as an eventby the event detection device 10, and lead to a decrease in accuracy ofobject recognition.

Therefore, the controller 30 performs control to raise the detectionthreshold for the event detection unit 63. By doing so, the controller30 performs control to change the detection sensitivity of the eventdetection being performed by the event detection device 10 (the gain ofthe event detection device 10) to such sensitivity that generation ofwater droplets or the like is not detected as noise. Even after thedetection threshold for the event detection is raised, the imaging bythe event detection device 10 is continued, and, in the event detectiondevice 10, an event detection process is performed with the detectionsensitivity changed by the controller 30.

The controller 30 again acquires a measurement value of the rain gauge(step S15), and then determines whether or not the measurement value ofthe rain gauge is smaller than the predetermined threshold (step S16).If the measurement value of the rain gauge is still equal to or greaterthan the predetermined threshold (NO in S16), the controller 30 returnsto step S15, and repeats the acquisition of a measurement value of therain gauge. If the measurement value of the rain gauge is smaller thanthe predetermined threshold (YES in S16), for example, the controller 30then determines that it has stopped raining, and performs control tolower the event detection threshold by a predetermined value or returnthe event detection threshold to the initial setting value (step S17).

After performing the control to lower the event detection threshold orreturn the event detection threshold to the initial setting value, thecontroller 30 determines whether or not the imaging by the eventdetection device 10 has ended (step S18). If the imaging has not endedyet (NO in S18), the controller 30 returns to step S11, and repeats theseries of processes described above. If the imaging by the eventdetection device 10 has ended (YES in S18), the controller 30 ends theseries of processes for event detection.

Control of the Event Detection Threshold

Here, a specific example of the event detection threshold control to beperformed in the process in step S14 or step S17 is described. It ispossible to control the event detection threshold for determining thegain of the event detection device 10, by changing the circuitconfiguration of the current-voltage conversion unit 631, thecapacitance ratio between the capacitive element 6331 and the capacitiveelement 6333 in the subtraction unit 633, or the threshold voltageV_(th) in the quantization unit 634.

Here, the control of the event detection threshold by changing thecircuit configuration of the current-voltage conversion unit 631 isspecifically described.

In the current-voltage conversion unit 631 having the configurationshown in FIG. 6, a photocurrent supplied from the light receiving unit61 through the signal input line L is converted into an optical voltagecorresponding to the logarithm of the photocurrent. The optical voltageis then output from the common connection node N₂ of the transistor 6312and the transistor 6315 to the buffer 632.

In a case where the switch element SW₁ and the switch element SW₃ enteran on-state or an off-state in the current-voltage conversion unit 631,the switch element SW₂ and the switch element SW₄ enter an on-state oran off-state, to change the detection threshold for the event detectionunit 63.

FIG. 10 shows, as a first circuit configuration, the substantial circuitconfiguration of the current-voltage conversion unit 631 in a case wherethe switch element SW₁ and the switch element SW₃ are in an on-statewhile the switch element SW₂ and the switch element SW₄ are in anoff-state.

Further, FIG. 11 shows, as a second circuit configuration, thesubstantial circuit configuration of the current-voltage conversion unit631 in a case where the switch element SW₁ and the switch element SW₃are in an off-state while the switch element SW₂ and the switch elementSW₄ are in an on-state.

In the case of the first circuit configuration in which the switchelement SW₁ and the switch element SW₃ are in an on-state while theswitch element SW₂ and the switch element SW₄ are in an off-state, thecurrent-voltage conversion unit 631 has a circuit configuration in whichthe transistor 6311 and the transistor 6314 are not cascade-connected,and the transistor 6313 and the transistor 6315 are notcascade-connected either, as shown in FIG. 10.

In the case of the second circuit configuration in which the switchelement SW₁ and the switch element SW₃ are in an off-state while theswitch element SW₂ and the switch element SW₄ are in an on-state, thecurrent-voltage conversion unit 631 has a circuit configuration in whichthe transistor 6311 and the transistor 6314 are cascade-connected, andthe transistor 6313 and the transistor 6315 are also cascade-connected,as shown in FIG. 11.

Here, the transistor 6311 and the transistor 6313, and the transistor6314 and the transistor 6315 are formed with field effect transistors ofthe same specification, for example. With this arrangement, the gain(the event detection threshold) of the event detection unit 63 in thecase of the second circuit configuration shown in FIG. 11 is about twiceas great as that in the case of the first circuit configuration shown inFIG. 10.

Therefore, in the process in step S14 in the event detection processaccording to Example 1 shown in FIG. 9, the circuit configuration of thecurrent-voltage conversion unit 631 is switched from the second circuitconfiguration shown in FIG. 11 to the first circuit configuration shownin FIG. 10, so that the event detection threshold, which is thedetection sensitivity of the event detection, can be lowered.

Note that, in the current-voltage conversion unit 631 having theconfiguration shown in FIG. 6, a circuit configuration in which thetransistors are cascade-connected in two stages has been described as anexample. However, this embodiment is not limited to two-stage cascadeconnection, and a circuit configuration in which the transistors arecascade-connected in three or more stages can be adopted.

Although an example case where the circuit configuration of thecurrent-voltage conversion unit 631 is switched so as to control theevent detection threshold has been described herein, it is also possibleto control the event detection threshold by changing the capacitanceratio between the capacitive element 6331 and the capacitive element6333 in the subtraction unit 633, or the threshold voltage V_(th) in thequantization unit 634.

Specifically, in the subtraction unit 633 having the configuration shownin FIG. 7, it is possible to control the event detection threshold bychanging the capacitance ratio (C₁/C₂) between the capacitive element6331 as the first capacitive element and the capacitive element 6333 asthe second capacitive element. Further, in the quantization unit 634having the configuration shown in FIG. 7, it is possible to control theevent detection threshold by adjusting the threshold voltage V_(th)serving as the inverting (−) input of the comparator 6341.

Example 2

Example 2 is an example in which the event detection threshold iscontrolled with the use of a measurement value of a rain gauge and noiseinformation in the imaging system 1A according to the first embodiment.An example flow in an event detection process according to Example 2 isshown in a flowchart in FIG. 12. The event detection process accordingto Example 2 is an example process according to a control method of thepresent disclosure, by which the detection sensitivity of the eventdetection being performed by the event detection device is controlled onthe basis of external information.

The controller 30 sets a predetermined initial setting value as theevent detection threshold, and causes the event detection device 10 toperform imaging (step S21). While imaging is being performed by theevent detection device 10, the controller 30 acquires a measurementvalue of the rain gauge provided as external information (step S22), andthen determines whether or not the measurement value of the rain gaugeis equal to or greater than a predetermined threshold (step S23). If themeasurement value is neither equal to nor greater than the predeterminedthreshold (NO in S23), the process returns to step S21, and the imagingby the event detection device 10 continues.

If the measurement value of the rain gauge is equal to or greater thanthe predetermined threshold (YES in S23), the controller 30 thendetermines whether or not the number of events in the plane (the numberof detected events) is equal to or larger than a predetermined threshold(step S24). If the number of events in the plane is neither equal to norlarger than the threshold (NO in S24), the controller 30 returns to stepS21, and causes the event detection device 10 to continue the imaging.Here, “in the plane” means within the imaging area corresponding to theangle of view of the event detection device 10 or within a specific areawithin the imaging area.

If the number of events in the plane is equal to or larger than thethreshold (YES in S24), the controller 30 performs control to raise thedetection threshold for the event detection unit 63 by a certain value,for example (step S25). That is, in a case where the measurement valueof the rain gauge is equal to or greater than the threshold, and thenumber of events in the plane is equal to or greater than the threshold,it is determined that there is a large amount of rain, and there is ahigh possibility that generation of a large number of water droplets orthe like will turn into noise for an object (a vehicle, a pedestrian, orthe like) originally desired to be detected as an event by the eventdetection device 10.

Therefore, the controller 30 performs control to raise the detectionthreshold for the event detection unit 63 (step S25), to change thedetection sensitivity of the event detection (the gain of the eventdetection device 10) to such sensitivity that generation of waterdroplets or the like is not detected as noise. Even after the eventdetection threshold is raised, the imaging by the event detection device10 is continued, and, in the event detection device 10, an eventdetection process is performed with the detection sensitivity changed bythe controller 30.

The controller 30 again acquires a measurement value of the rain gauge(step S26), and then determines whether or not the measurement value ofthe rain gauge is smaller than the predetermined threshold (step S27).If the measurement value of the rain gauge is still equal to or greaterthan the predetermined threshold (NO in S27), the controller 30 returnsto step S26, and repeats the acquisition of a measurement value of therain gauge. If the controller 30 determines the measurement value of therain gauge to be smaller than the predetermined threshold (YES in S27),the controller 30 then performs control to lower the event detectionthreshold by a predetermined value or return the event detectionthreshold to the initial setting value (step S28).

After performing the control to lower the event detection threshold orreturn the event detection threshold to the initial setting value, thecontroller 30 determines whether or not the imaging by the eventdetection device 10 has ended (step S29). If the imaging has not endedyet (NO in S29), the controller 30 returns to step S21, and repeats theseries of processes described above. If the imaging by the eventdetection device 10 has ended (YES in S29), the controller 30 ends theseries of processes for event detection.

Example Configuration of an Object Recognition System According to theFirst Embodiment

Next, an object recognition system according to the first embodimentthat performs object recognition using the imaging system 1A accordingto the first embodiment having the above configuration is described.Like the imaging system 1A according to the first embodiment, an objectrecognition system according to the first embodiment can be mounted on amobile unit such as a vehicle and be used for object recognition in anevent.

FIG. 13 is a block diagram showing an example system configuration of anobject recognition system according to the first embodiment.

As shown in FIG. 13, an object recognition system 2A according to thefirst embodiment includes a recognition processing unit 60, in additionto the event detection device 10, the controller 30, the data processingunit 40, and the image recording unit 50 in the imaging system 1Aaccording to the first embodiment shown in FIG. 1. Details of the eventdetection device 10, the controller 30, the data processing unit 40, andthe image recording unit 50 are as described above.

In the object recognition system 2A according to the first embodimenthaving the configuration described above, event data processed by thedata processing unit 40 is supplied to the recognition processing unit60. The recognition processing unit 60 performs a process of objectrecognition in an event, on the basis of the event data supplied fromthe data processing unit 40. In the object recognition by therecognition processing unit 60, it is possible to use a known patternrecognition technique, such as a technique for performing imagerecognition by comparing the feature points of an image provided asteacher data with the feature points of a captured image of the object,for example.

Example 3

Example 3 is an example in which an object recognition result isreflected by the control of the event detection threshold in the objectrecognition system 2A according to the first embodiment. An example flowin an event detection process according to Example 3 is shown in aflowchart in FIG. 14. The event detection process according to Example 3is an example process according to a control method of the presentdisclosure, by which the detection sensitivity of the event detectionbeing performed by the event detection device is controlled on the basisof external information.

The controller 30 sets a predetermined initial setting value as theevent detection threshold, and causes the event detection device 10 toperform imaging (step S31). While imaging is being performed by theevent detection device 10, the controller 30 acquires a measurementvalue of the rain gauge provided as external information (step S32), andthen determines whether or not the measurement value of the rain gaugeis equal to or greater than a predetermined threshold (step S33). If themeasurement value is neither equal to nor greater than the predeterminedthreshold (NO in S33), the process returns to step S31, and the imagingby the event detection device 10 continues.

If the measurement value of the rain gauge is equal to or greater thanthe predetermined threshold (YES in S33), the controller 30 performscontrol to raise the event detection threshold (step S34), to change thedetection sensitivity for event detection to such sensitivity thatgeneration of water droplets or the like is not detected as noise. Evenafter the event detection threshold is raised, the imaging by the eventdetection device 10 is continued, and, in the event detection device 10,event detection is performed with the detection sensitivity changed bythe controller 30.

Next, the controller 30 performs an object recognition process on thebasis of event data indicating the occurrence of an event output fromthe event detection device 10 (step S35), and then determines whether ornot the object recognition has been successfully completed (step S36). Aspecific example of the object recognition process in step S35 will bedescribed later. Whether or not the object recognition has beensuccessfully completed can be determined depending on whether or not theregion of the vehicle with respect to the angle of view is equal to orlarger than a certain proportion, for example.

If the object recognition has failed (NO in S36), the controller 30returns to step S34, and performs control to raise the event detectionthreshold. That is, a loop process from step S34 to step S35 to step S36to step S34 is performed, so that control is performed to raise theevent detection threshold to a value within the range in which theobject can be recognized. As the event detection threshold is raised,the influence of noise caused by water droplets or the like can bereduced.

If the object recognition has been successfully completed (YES in S36),the controller 30 acquires a measurement value of the rain gauge (stepS37), and then determines whether or not the measurement value of therain gauge is smaller than the predetermined threshold (step S38). Ifthe measurement value of the rain gauge is still equal to or greaterthan the predetermined threshold (NO in S38), the controller 30 returnsto step S37, and repeats the acquisition of a measurement value of therain gauge. If the controller 30 determines the measurement value of therain gauge to be smaller than the predetermined threshold (YES in S38),the controller 30 then performs control to lower the event detectionthreshold by a predetermined value or return the event detectionthreshold to the initial setting value (step S39).

After performing the control to lower the event detection threshold orreturn the event detection threshold to the initial setting value, thecontroller 30 determines whether or not the imaging by the eventdetection device 10 has ended (step S40). If the imaging has not endedyet (NO in S40), the controller 30 returns to step S41, and repeats theseries of processes described above. If the imaging by the eventdetection device 10 has ended (YES in S40), the controller 30 ends theseries of processes for event detection.

FIG. 15 is a flowchart showing an example process in a specific exampleof the object recognition process in step S35.

The controller 30 detects an object, such as a vehicle, for example,using information (event data indicating the occurrence of an event, forexample) from the event detection device 10 (step S41), and thenperforms process of specifying the area can be detected as the vehiclewithin the angle of view (step S42). Next, the controller 30 determineswhether or not the area that can be detected as the vehicle is equal toor larger than a predetermined threshold (step S43). If the area issmaller than the predetermined threshold (NO in S43), the controller 30returns to the process in step S42. If the area is equal to or largerthan the predetermined threshold (YES in S43), the controller 30recognizes the area as the vehicle (object), returns to the flow shownin FIG. 14, and proceeds to the process in step S36.

Note that Example 3 described above is an example case where a techniquefor causing an object recognition result to be reflected by the controlof the event detection threshold is applied to Example 1 in which theevent detection threshold is controlled on the basis of a measurementvalue of a rain gauge. However, the technique of Example 3 can also beapplied to Example 2 in which the event detection threshold iscontrolled with the use of a measurement value of the rain gauge andnoise information.

Second Embodiment of the Present Disclosure Example Configuration of anImaging System According to the Second Embodiment

FIG. 16 is a block diagram showing an example system configuration of animaging system according to a second embodiment of the presentdisclosure.

As shown in FIG. 16, an imaging system 1B according to the secondembodiment of the present disclosure includes an event detection device10, an imaging device 20, a controller 30, a data processing unit 40,and an image recording unit 50. The event detection device 10, thecontroller 30, the data processing unit 40, and the image recording unit50 are as explained above in the description of the imaging system 1Aaccording to the first embodiment.

The imaging device 20 can be a synchronous imaging device that performsimaging at a fixed frame rate in synchronization with a verticalsynchronization signal, and outputs image data in a frame format. Notethat the synchronous imaging device can be a complementary metal oxidesemiconductor (CMOS) image sensor, a charge coupled device (CCD) imagesensor, or the like, for example.

Example Configuration of the Imaging Device

The basic configuration of the imaging device 20 in the imaging system1B according to the second embodiment is now described. In thisexplanation, the imaging device 20 is a CMOS image sensor that is a kindof X-Y address imaging device, for example. A CMOS image sensor is animage sensor manufactured by applying a CMOS process or partially usinga CMOS process. However, the imaging device 20 is not necessarily a CMOSimage sensor.

Example Configuration of a CMOS Image Sensor

FIG. 17 is a block diagram schematically showing the configuration of aCMOS image sensor that is an example of the imaging device 20 in theimaging system 1B according to the second embodiment of the presentdisclosure.

The imaging device 20 according to this example includes a pixel arrayunit 22 in which pixels 21 each including a light receiving unit(photoelectric conversion unit) are two-dimensionally arranged in therow direction and the column direction, or in a matrix, and a peripheralcircuit unit of the pixel array unit 22. Here, the row direction refersto the array direction of the pixels 21 in the pixel rows, and thecolumn direction refers to the array direction of the pixels 21 in thepixel columns. A pixel 21 performs photoelectric conversion to generateand accumulate photoelectric charge corresponding to the amount ofreceived light.

The imaging device 20 according to this example is an RGB sensor inwhich red (R), green (G), and blue (B) color filters are incorporated inthe respective pixels 21 of the pixel array unit 22, for example.However, the imaging device 20 is not necessarily an RGB sensor.

The peripheral circuit unit of the pixel array unit 22 includes a rowselection unit 23, a constant-current supply unit 24, an analog-digitalconversion unit 25, a horizontal transfer scanning unit 26, a signalprocessing unit 27, and a timing control unit 28, for example.

In the pixel array unit 22, pixel drive lines 31 ₁ to 31 _(m)(hereinafter collectively referred to as the “pixel drive lines 31” insome cases) are provided in the row direction for the respective pixelrows in the matrix-like pixel array. Also, vertical signal lines 32 ₁ to32 _(n) (hereinafter collectively referred to as the “vertical signallines 32” in some cases) are provided the column direction for therespective pixel columns. The pixel drive lines 31 transmit drivesignals for performing driving when signals are read from the pixels 21.In FIG. 1, each pixel drive line 31 is shown as one wiring line, but isnot necessarily one wiring line. One end of each pixel drive line 31 isconnected to the output end of the row selection unit 23 correspondingto the respective rows.

In the description below, the respective circuit portions of theperipheral circuit unit of the pixel array unit 22, which are the rowselection unit 23, the constant-current supply unit 24, theanalog-digital conversion unit 25, the horizontal transfer scanning unit26, the signal processing unit 27, and the timing control unit 28, areexplained.

The row selection unit 23 is formed with a shift register, an addressdecoder, and the like, and controls scanning of a pixel row and theaddress of the pixel row, when selecting each pixel 21 of the pixelarray unit 22. The configuration of this row selection unit 23 is notspecifically shown in the drawing, but normally has a configuration thatincludes two scanning systems: a read scanning system and a sweepscanning system.

To read pixel signals from the pixels 21, the read scanning systemsequentially selects and scans each pixel 21 in the pixel array unit 22row by row. The pixel signals to be read from the pixels 21 are analogsignals. The sweep scanning system performs sweep scanning on the readrow on which read scanning is to be performed by the read scanningsystem, prior to the read scanning by the time equivalent to the shutterspeed.

Through the sweep scanning by this sweep scanning system, unnecessaryelectric charges are swept out of the light receiving units(photoelectric conversion units) of the pixels 21 of the read row, andthus, the light receiving units are reset. As the unnecessary electriccharges are swept (reset) by the sweep scanning system, a so-calledelectronic shutter operation is then performed. Here, an electronicshutter operation is an operation to discard photoelectric charges ofthe light receiving units, and newly start exposure (start accumulatingphotoelectric charges).

The constant-current supply unit 24 includes a plurality of currentsources I (see FIG. 18) each including a MOS transistor and connected toeach of the vertical signal lines 32 ₁ to 32 _(n) in the respectivepixel columns, for example, and supplies a bias current to each pixel 21of the pixel row selectively scanned by the row selection unit 23through each of the vertical signal lines 32 ₁ to 32 _(n).

The analog-digital conversion unit 25 includes a set of a plurality ofanalog-digital converters provided in accordance with the pixel columnsof the pixel array unit 22 (or provided for the respective pixelcolumns, for example). The analog-digital conversion unit 25 is acolumn-parallel analog-digital conversion unit that converts analogpixel signals output through the respective vertical signal lines 32 ₁to 32 _(n) for the respective pixel columns into digital signals.

Each analog-digital converter in the column-parallel analog-digitalconversion unit 25 can be a single-slope analog-digital converter thatis an example of a reference-signal-comparison analog-digital converter,for example. However, each analog-digital converter is not necessarily asingle-slope analog-digital converter, but can be asequential-comparison analog-digital converter, a delta-sigma-modulation(AZ-modulation) analog-digital converter, or the like.

This example of the analog-digital converters in the column-parallelanalog-digital conversion unit 25 also applies to the analog-digitalconverters in the analog-digital conversion unit forming the columnprocessing unit 15 (see FIG. 2) of the event detection device 10described above.

The horizontal transfer scanning unit 26 is formed with a shiftregister, an address decoder, and the like, and controls scanning of apixel column and the address of the pixel column, when reading a signalfrom each pixel 21 in the pixel array unit 22. Under the control of thehorizontal transfer scanning unit 26, the pixel signals converted intodigital signals by the analog-digital conversion unit 25 are read out toa horizontal transfer line (horizontal output line) 29 on a pixel columnbasis.

The signal processing unit 27 performs predetermined signal processingon the digital pixel signals supplied through the horizontal transferline 29, to generate two-dimensional image data. For example, the signalprocessing unit 27 corrects a vertical line defect or a point defect,clamps signals, or performs digital signal processing such asparallel-serial conversion, compression, encoding, adding, averaging,and intermittent operation. The signal processing unit 27 outputs thegenerated image data as an output signal of the imaging device 20, to adevice in a later stage.

The timing control unit 28 generates various kinds of timing signals,clock signals, control signals, and the like, on the basis of a verticalsynchronization signal VD and a horizontal synchronization signal HDsupplied from the outside, as well as a master clock MCK (not shown) andthe like. On the basis of these generated signals, the timing controlunit 28 then performs drive control on the row selection unit 23, theconstant-current supply unit 24, the analog-digital conversion unit 25,the horizontal transfer scanning unit 26, the signal processing unit 27,and the like.

Under the control of the timing control unit 28, the imaging device 20performs imaging in synchronization with a synchronization signal suchas the vertical synchronization signal VD. That is, the imaging device20 is a synchronous imaging device that performs imaging at a fixedframe rate.

Example Circuit Configuration of a Pixel

FIG. 18 is a circuit diagram showing an example circuit configuration ofa pixel 21 of the pixel array unit 22 in the imaging device 20.

The pixel 21 includes a photodiode 211 as a light receiving unit(photoelectric conversion unit), for example. The pixel 21 has a pixelconfiguration that includes a transfer transistor 212, a resettransistor 213, an amplification transistor 214, and a selectiontransistor 215, in addition to the photodiode 211.

Note that, although N-type MOS transistors are used as the fourtransistors of the transfer transistor 212, the reset transistor 213,the amplification transistor 214, and the selection transistor 215 inthis example, the combination of the conductivity types of the fourtransistors 212 to 215 shown herein is merely an example, and is notlimited to this combination.

For this pixel 21, a plurality of pixel drive lines is provided as theabove pixel drive lines 31 for the respective pixels 21 in the samepixel row. The plurality of pixel drive lines is connected, on a pixelrow basis, to the output ends of the row selection unit 23 correspondingto the respective pixel rows. The row selection unit 23 outputs atransfer signal TRG, a reset signal RST, and a selection signal SEL tothe plurality of pixel drive lines, as appropriate.

In the photodiode 211, the anode electrode is connected to a powersupply on the lower potential side (the ground, for example), receivedlight (incident light) is photoelectrically converted into photoelectriccharge (photoelectrons herein) with a charge amount corresponding to thelight amount, and the photoelectric charge is accumulated. The cathodeelectrode of the photodiode 211 is electrically connected to the gateelectrode of the amplification transistor 214 via the transfertransistor 212. Here, the region to which the gate electrode of theamplification transistor 214 is electrically connected is a floatingdiffusion (floating diffusion region/impurity diffusion region) FD. Thefloating diffusion FD is a charge-voltage conversion unit that convertselectric charge into voltage.

The transfer signal TRG that is active at the high level (the V_(DD)level, for example) is supplied from the row selection unit 23 to a gateelectrode of the transfer transistor 212. The transfer transistor 212enters an on-state in response to the transfer signal TRG, and transfersthe photoelectric charge that has been photoelectrically converted bythe photodiode 211 and is accumulated in the photodiode 211, to thefloating diffusion FD.

The reset transistor 213 is connected between the power supply line ofthe power supply voltage V_(DD) and the floating diffusion FD. To thegate electrode of the reset transistor 213, the reset signal RST that ishigh at the high level is supplied from the row selection unit 23. Thereset transistor 213 enters an on-state in response to the reset signalRST, and resets the floating diffusion FD by discarding the electriccharge in the floating diffusion FD to a node of the power supplyvoltage V_(DD).

The gate electrode of the amplification transistor 214 is connected tothe floating diffusion FD, and the drain electrode is connected to thepower supply line of the power supply voltage V_(DD). The amplificationtransistor 214 serves as an input unit of a source follower that reads asignal obtained through photoelectric conversion performed by thephotodiode 211. The source electrode of the amplification transistor 214is connected to the vertical signal line 32 via the selection transistor215. The amplification transistor 214 and the current source I connectedto one end of the vertical signal line 32 then constitute a sourcefollower that converts the voltage of the floating diffusion FD into thepotential of the vertical signal line 32.

The selection transistor 215 has its drain electrode connected to thesource electrode of the amplification transistor 214, and its sourceelectrode connected to the vertical signal line 32. The selection signalSEL that is active at the high level is supplied from the row selectionunit 23 to the gate electrode of the selection transistor 215. Theselection transistor 215 enters an on-state in response to the selectionsignal SEL, to put the pixel 21 into a selected state and transmit asignal output from the amplification transistor 214 to the verticalsignal line 32.

Note that a 4Tr configuration formed with the four transistors (Tr) ofthe transfer transistor 212, the reset transistor 213, the amplificationtransistor 214, and the selection transistor 215 has been described asan example of the pixel circuit of a pixel 21, but this embodiment isnot limited this example. For example, the selection transistor 215 maybe excluded, and the amplification transistor 214 may be provided withthe functions of the selection transistor 25 so that a 3Tr configurationis formed. If necessary, the number of transistors may be increased toform a 5Tr or higher configuration.

Example Configuration of a Chip Structure

The chip (semiconductor integrated circuit) structure of the imagingdevice 20 having the above configuration can be a flat chip structure ora stacked chip structure, for example. In the imaging device 20 havingeither a flat chip structure or a stacked chip structure, when thesubstrate face on which the wiring layers are disposed is the frontsurface (the front face), the pixel 21 can have a front-illuminatedpixel structure that captures light emitted from the front surface side,or can have a back-illuminated pixel structure that captures lightemitted from the back surface side, which is the opposite side from thefront surface side. In the description below, a flat chip structure anda stacked chip structure are explained.

(Flat Chip Structure)

FIG. 19 is a plan view schematically showing a flat chip structure ofthe imaging device 20.

As shown in FIG. 19, a flat chip structure (a so-called flat structure)is a structure in which the circuit portions around the pixel array unit22 are formed on the same semiconductor substrate 201 as that of thepixel array unit 22 in which the pixels 21 are arranged in a matrix.Specifically, the row selection unit 23, the constant-current supplyunit 24, the analog-digital conversion unit 25, the horizontal transferscanning unit 26, the signal processing unit 27, the timing control unit28, and the like are formed on the same semiconductor substrate 201 asthat of the pixel array unit 22.

(Stacked Chip Structure)

FIG. 20 is an exploded perspective view schematically showing a stackedchip structure of the imaging device 20.

As shown in FIG. 20, a stacked chip structure (a so-called stackstructure) is a structure in which at least two semiconductor substratesincluding a first semiconductor substrate 202 and a second semiconductorsubstrate 203 are stacked. In this stack structure, the pixel array unit22 is formed on the first semiconductor substrate 202 that is the firstlayer. Further, the circuit portions such as the row selection unit 23,the constant-current supply unit 24, the analog-digital conversion unit25, the horizontal transfer scanning unit 26, the signal processing unit27, and the timing control unit 28 are formed on the secondsemiconductor substrate 203 that is the second layer. Further, the firstsemiconductor substrate 202 as the first layer and the secondsemiconductor substrate 203 as the second layer are electricallyconnected through connecting portions 33A and 33B such as vias or Cu—Cujoints.

With the imaging device 20 having this stack structure, a processsuitable for manufacturing the pixels 21 can be applied to the firstsemiconductor substrate 202 as the first layer, and a process suitablefor manufacturing the circuit portions can be applied to the secondsemiconductor substrate 203 as the second layer. Thus, the process canbe optimized in manufacturing the imaging device 20. In particular, astate-of-the-art process can be adopted in manufacturing the circuitportions.

Note that, although a stack structure having a two-layer structureformed by stacking the first semiconductor substrate 202 and the secondsemiconductor substrate 203 has been described as an example, the stackstructure is not necessarily a two-layer structure, but may be astructure formed with three or more layers. Further, in the case of astack structure formed with three or more layers, the circuit portionssuch as the row selection unit 23, the constant-current supply unit 24,the analog-digital conversion unit 25, the horizontal transfer scanningunit 26, and the signal processing unit 27 can be formed in adistributed manner on the semiconductor substrates of the second andsubsequent layers.

Example System Configuration of an Object Recognition System Accordingto the Second Embodiment

Next, an object recognition system according to the second embodimentthat performs object recognition using the imaging system 1B accordingto the second embodiment having the above configuration is described.Like the imaging system 1B according to the second embodiment, an objectrecognition system according to the second embodiment can be mounted ona mobile unit such as a vehicle and be used for object recognition in anevent.

FIG. 21 is a block diagram showing an example system configuration of anobject recognition system according to the second embodiment.

As shown in FIG. 21, an object recognition system 2B according to thesecond embodiment includes a recognition processing unit 60, in additionto the event detection device 10, the imaging device 20, the controller30, the data processing unit 40, and the image recording unit 50 in theimaging system 1B according to the second embodiment shown in FIG. 16.Details of the event detection device 10, the imaging device 20, thecontroller 30, the data processing unit 40, and the image recording unit50 are as described above.

In the object recognition system 2B according to the second embodimenthaving the above configuration, event data that is output from the eventdetection device 10, and image data that is output from the imagingdevice 20 are subjected to predetermined data processing in the dataprocessing unit 40, and are then supplied to the recognition processingunit 60. The recognition processing unit 60 performs a process of objectrecognition in an event, on the basis of the event data or the imagedata supplied from the data processing unit 40. In the objectrecognition by the recognition processing unit 60, it is possible to usea known pattern recognition technique, such as a technique forperforming image recognition by comparing the feature points of an imageprovided as teacher data with the feature points of a captured image ofthe object, for example.

In the description below, a specific example of event detection to beperformed in the object recognition system 2B according to the secondembodiment having the above configuration will be explained. In theobject recognition system 2B according to the second embodiment, anobject recognition process for an object such as a vehicle is performedon the basis of event data that is output from the event detectiondevice 10, and image data that is output from the imaging device 20.

Note that the event detection device 10 formed with an asynchronousimaging device has a pixel configuration including an event detectionunit 63. Therefore, the pixel size in the event detection device 10 hasto be larger than that in the synchronous imaging device 20, andtherefore, the resolution of the event detection device 10 is lower thanthat of the imaging device 20 that performs imaging at a fixed framerate. On the other hand, the imaging device 20 formed with a synchronousimaging device has a higher resolution than that of an asynchronousimaging device.

Example 4

Example 4 is an example in which an object recognition result isreflected by the control of the event detection threshold in the objectrecognition system 2B according to the second embodiment. An exampleflow in an event detection process according to Example 4 is shown in aflowchart in FIG. 22.

The controller 30 sets a predetermined initial setting value as theevent detection threshold, and causes the event detection device 10 toperform imaging (step S51). While imaging is being performed by theevent detection device 10, the controller 30 acquires a measurementvalue of the rain gauge provided as external information (step S52), andthen determines whether or not the measurement value of the rain gaugeis equal to or greater than a predetermined threshold (step S53). If themeasurement value is neither equal to nor greater than the predeterminedthreshold (NO in S53), the process returns to step S51, and the imagingby the event detection device 10 continues.

If the measurement value of the rain gauge is equal to or greater thanthe predetermined threshold (YES in S53), the controller 30 performscontrol to raise the event detection threshold (step S54), to change thedetection sensitivity for event detection to such sensitivity thatgeneration of water droplets or the like is not detected as noise. Evenafter the event detection threshold is raised, the imaging by the eventdetection device 10 is continued, and, in the event detection device 10,event detection is performed with the detection sensitivity changed bythe controller 30.

Next, the controller 30 performs an object recognition process fordetermining whether or not the region of the vehicle with respect to theangle of view is equal to or larger than a certain proportion (stepS55), and then determines whether or not the object recognition has beensuccessfully completed (step S56). If the object recognition fails (NOin S56), the controller 30 returns to step S54, and performs control toraise the event detection threshold. That is, a loop process from stepS54 to step S55 to step S56 to step S54 is performed, so that control isperformed to raise the event detection threshold to a value within therange in which the object can be recognized. As the event detectionthreshold is raised, the influence of noise caused by water droplets orthe like can be reduced. A specific example of the object recognitionprocess in step S55 will be described later.

If the object recognition has been successfully completed (YES in S56),the controller 30 acquires a measurement value of the rain gauge (stepS57), and then determines whether or not the measurement value of therain gauge is smaller than the predetermined threshold (step S58). Ifthe measurement value of the rain gauge is still equal to or greaterthan the predetermined threshold (NO in S58), the controller 30 returnsto step S57, and repeats the acquisition of a measurement value of therain gauge. If the controller 30 determines that the measurement valueof the rain gauge is smaller than the predetermined threshold, and theweather has recovered (YES in S58), the controller 30 then performscontrol to lower the event detection threshold or return the eventdetection threshold to the initial setting value (step S59).

After performing the control to lower the event detection threshold orreturn the event detection threshold to the initial setting value, thecontroller 30 determines whether or not the imaging by the eventdetection device 10 has ended (step S60). If the imaging has not endedyet (NO in S60), the controller 30 returns to step S51, and repeats theseries of processes described above. If the imaging by the eventdetection device 10 has ended (YES in S60), the controller 30 ends theseries of processes for event detection.

FIG. 23 is a flowchart showing a specific example of the objectrecognition process in step S55.

The controller 30 determines whether or not the object, which is avehicle, for example, can be detected only with information from theevent detection device 10 formed with an asynchronous imaging device(step S61). If the object can be detected (YES in S61), the object,which is a vehicle, for example, is detected with the use of information(event data indicating the occurrence of an event, for example) from theevent detection device 10 (step S62).

Next, the controller 30 specifies the area that can be detected as thevehicle within the angle of view (step S63). If it is not possible todetect the vehicle only with the event detection device 10 (NO in S61),the controller 30 detects the vehicle, using the information from theevent detection device 10 and information from the imaging device 20having a higher resolution than that of the event detection device 10(step S64), and after that, proceeds to the process in step S63.

Next, the controller 30 determines whether or not the area that can bedetected as the vehicle is equal to or larger than a predeterminedthreshold (step S65). If the area is smaller than the predeterminedthreshold (NO in S65), the controller 30 returns to the process in stepS63. If the area is equal to or larger than the predetermined threshold(YES in S65), the controller 30 recognizes the area as the vehicle(object), returns to the flow shown in FIG. 22, and proceeds to theprocess in step S56.

As described above, in a case where a vehicle cannot be detected onlywith the event detection device 10 in an object recognition process, theimaging device 20 that has a higher resolution than that of anasynchronous imaging device is used. Thus, an object recognition processcan be performed with higher accuracy, even in the case of bad weathersuch as rainy weather, for example.

Example 5

Example 5 is an example in which the event detection threshold iscontrolled with the use of a measurement value of a rain gauge and noiseinformation in the object recognition system 2B according to the secondembodiment. An example flow in an event detection process according toExample 5 is shown in a flowchart in FIG. 24.

The controller 30 sets a predetermined initial setting value as theevent detection threshold, and causes the event detection device 10 toperform imaging (step S71). While imaging is being performed by theevent detection device 10, the controller 30 acquires a measurementvalue of the rain gauge provided as external information (step S72), andthen determines whether or not the measurement value of the rain gaugeis equal to or greater than a predetermined threshold (step S73). If themeasurement value is neither equal to nor greater than the predeterminedthreshold (NO in S73), the process returns to step S71, and the imagingby the event detection device 10 continues.

If the measurement value of the rain gauge is equal to or greater thanthe predetermined threshold (YES in S73), the controller 30 thendetermines whether or not the number of events (the number of detectedevents) in the plane of the front windshield, for example, is equal toor larger than a predetermined threshold (step S74). If the number ofevents in the plane is neither equal to nor larger than the threshold(NO in S74), the controller 30 returns to step S71, and causes the eventdetection device 10 to continue the imaging.

If the number of events in the plane is equal to or larger than thethreshold (YES in S74), the controller 30 performs control to raise theevent detection threshold (step S75). That is, in a case where themeasurement value of the rain gauge is equal to or greater than thethreshold, and the number of events in the plane is equal to or greaterthan the threshold, it is determined that there is a large amount ofrain, and there is a high possibility that generation of a large numberof water droplets or the like will turn into noise for an object (avehicle, a pedestrian, or the like) originally desired to be detected asan event by the event detection device 10.

Therefore, the controller 30 performs control to raise the eventdetection threshold (step S75), to change the detection sensitivity ofthe event detection to such sensitivity that generation of waterdroplets or the like is not detected as noise. Even after the eventdetection threshold is raised, the imaging by the event detection device10 is continued, and, in the event detection device 10, an eventdetection process is performed with the detection sensitivity changed bythe controller 30.

Next, the controller 30 performs an object recognition process (theobject recognition process in FIG. 23) (step S76), and then determineswhether or not the object recognition has been successfully completed(step S77). If the object recognition has failed, or if the region ofthe vehicle with respect to the angle of view is smaller than a certainproportion (NO in S77), the controller 30 returns to step S75, andperforms control to raise the event detection threshold. That is, a loopprocess from step S75 to step S76 to step S77 to step S75 is performed,so that control is performed to raise the event detection threshold to avalue within the range in which the object can be recognized. As theevent detection threshold is raised, the influence of noise caused bywater droplets or the like can be reduced.

If the object recognition has been successfully completed, or if theregion of the vehicle with respect to the angle of view is equal to orlarger than the certain proportion (YES in S77), the controller 30acquires a measurement value of the rain gauge (step S78), and thendetermines whether or not the measurement value of the rain gauge issmaller than the predetermined threshold (step S79). If the measurementvalue of the rain gauge is still equal to or greater than thepredetermined threshold (NO in S79), the controller 30 returns to stepS78, and repeats the acquisition of a measurement value of the raingauge. If the controller 30 determines that the measurement value of therain gauge is smaller than the predetermined threshold, and the weatherhas recovered (YES in S79), the controller 30 then performs control tolower the event detection threshold or return the event detectionthreshold to the initial setting value (step S80).

After performing the control to lower the event detection threshold orreturn the event detection threshold to the initial setting value, thecontroller 30 determines whether or not the imaging by the eventdetection device 10 has ended (step S81). If the imaging has not endedyet (NO in S81), the controller 30 returns to step S71, and repeats theseries of processes described above. If the imaging by the eventdetection device 10 has ended (YES in S81), the controller 30 ends theseries of processes for event detection.

Example 6

Example 6 is an example in which an object recognition process isperformed twice before and after the control of the event detectionthreshold in the object recognition system 2B according to the secondembodiment. An example flow in an event detection process according toExample 6 is shown in a flowchart in FIG. 25.

The controller 30 sets a predetermined initial setting value as theevent detection threshold, and causes the event detection device 10 toperform imaging (step S91). While imaging is being performed by theevent detection device 10, the controller 30 acquires a measurementvalue of the rain gauge provided as external information (step S92), andthen determines whether or not the measurement value of the rain gaugeis equal to or greater than a predetermined threshold (step S93). If themeasurement value is neither equal to nor greater than the predeterminedthreshold (NO in S93), the process returns to step S91, and the imagingby the event detection device 10 continues.

If the measurement value of the rain gauge is equal to or greater thanthe predetermined threshold (YES in S 93), the controller 30 performs anobject recognition process (the object recognition process in FIG. 23)(step S95). If the object recognition has been successfully completed,or if the region of the vehicle with respect to the angle of view isequal to or greater than a certain proportion (NO in S95), thecontroller 30 determines that object recognition is possible even in asituation where there is a large amount of rain. The controller 30 thenreturns to step S91, and repeats the processes described above.

If the object recognition fails (YES in S95), the controller 30determines that it is difficult to perform the object recognitionbecause of the large amount of rain, and performs control to raise theevent detection threshold (step S96). Even after the event detectionthreshold is raised, the imaging by the event detection device 10 iscontinued, and, in the event detection device 10, an event detectionprocess is performed with the detection sensitivity changed by thecontroller 30.

After raising the event detection threshold, the controller 30 againperforms the object recognition process (the object recognition processin FIG. 23) (step S97), and then determines whether or not the objectrecognition has been successfully completed (step S98). If the objectrecognition has failed, or if the region of the vehicle with respect tothe angle of view is smaller than the certain proportion (NO in S98),the controller 30 returns to step S96, and performs control to raise theevent detection threshold. That is, a loop process from step S96 to stepS97 to step S98 to step S96 is performed, so that control is performedto raise the event detection threshold to a value within the range inwhich the object can be recognized. As the event detection threshold israised, the influence of noise caused by water droplets or the like canbe reduced.

If the object recognition has been successfully completed (YES in S98),the controller 30 acquires a measurement value of the rain gauge (stepS99), and then determines whether or not the measurement value of therain gauge is smaller than the predetermined threshold (step S100). Ifthe measurement value of the rain gauge is still equal to or greaterthan the predetermined threshold (NO in S100), the controller 30 returnsto step S99, and repeats the acquisition of a measurement value of therain gauge. If the controller 30 determines that the measurement valueof the rain gauge is smaller than the predetermined threshold, and theweather has recovered (YES in S100), the controller 30 then performscontrol to lower the event detection threshold or return the eventdetection threshold to the initial setting value (step S101).

After performing the control to lower the event detection threshold orreturn the event detection threshold to the initial setting value, thecontroller 30 determines whether or not the imaging by the eventdetection device 10 has ended (step S102). If the imaging has not endedyet (NO in S102), the controller 30 returns to step S91, and repeats theseries of processes described above. If the imaging by the eventdetection device 10 has ended (YES in S102), the controller 30 ends theseries of processes for event detection.

Example 7

While a vehicle is traveling, there is a plurality of objects to bedetected as events by the event detection device 10 in some cases. Whilea vehicle is traveling in rainy weather, there might be adhesion of manywater droplets or adhesion of no water droplets, depending on regions onthe front windshield. FIG. 26A shows a situation where there is aplurality of objects to be detected as events. FIG. 26B shows asituation where there is adhesion of many water droplets or adhesion ofno water droplets.

FIG. 26A illustrates an example case where the region in which an eventis to be detected is divided into three regions: a region A including avehicle traveling forward, a region B including a pedestrian walking onthe left side in the traveling direction, and a region C including amotorcycle traveling on the left side in the traveling direction. FIG.26B illustrates an example case where the subject region is divided intothree regions: a region A in which many water droplets adhere to thefront windshield, a region B in which some water droplets adhere to thefront windshield, and a region C in which very few water droplets adhereto the front windshield or no water droplets adhere to the frontwindshield.

Example 7 is an example in which the event detection threshold iscontrolled region by region in a situation illustrated in FIG. 26A or26B. An example flow in an event detection process according to Example7 is shown in a flowchart in FIG. 27.

For example, in the object recognition system 2B according to the secondembodiment shown in FIG. 21, the controller 30 first performs a processof determining regions (step S111). In the region determination processin step S111, the region A including a vehicle, the region B including apedestrian, and the region C including a motorcycle in FIG. 26A can bedetermined by a known pattern recognition technique, such as a techniquefor performing image recognition by comparing the feature points of animage provided as teacher data with the feature points of a capturedimage of the object, for example. Meanwhile, the region A to which alarge number of water droplets adhere, the region B to which a smallnumber of water droplets adhere, and the region C to which very fewwater droplets adhere as shown in FIG. 26B can be determined on thebasis of the number of events that are water droplets detected as eventsby the event detection device 10, for example.

Next, the controller 30 determines the region to be subjected to eventdetection, on the basis of the result of the region determinationprocess performed in step S111 (step S112). The controller 30 thenperforms an event detection process suitable for the region A in thecase of the region A (step S113), performs an event detection processsuitable for the region B in the case of the region B (step S114), andperforms an event detection process suitable for the region C in thecase of the region C (step S115).

For example, the event detection processes according to Examples 4 to 6described above can be applied to the event detection processes in stepsS 113, S 114, and S 115. The control of the event detection thresholdsin the region A to which a large number of water droplets adhere, theregion B to which a small number of water droplets adhere, and theregion C to which very few water droplets adhere as shown in FIG. 26B isnow described, for example. In this case, the initial setting value ofthe event detection threshold is set at the values corresponding to theregions A, B, and C, for example, so that event detection thresholdcontrol suitable for the respective regions can be performed. As aresult, for example, it is possible to increase the accuracy ofrecognition of an object such as a vehicle, a pedestrian, or amotorcycle included in the region A to which many water droplets adhere.Note that, as another example of an event detection process, the regiondetermination process may be performed after determination as to whetheror not the weather is bad, in the event detection processes according toExamples 4 to 6 described above.

Modifications

Although the technology according to the present disclosure has beendescribed so far on the basis of the preferred embodiments, thetechnology according to the present disclosure is not limited to theseembodiments. The configurations and structures of the imaging systemsand the object recognition systems described in the above embodimentsare examples, and can be modified. For example, in the aboveembodiments, the pixel signal generation unit 62 is provided for eachlight receiving unit 61 to form the pixels 11. However, it is alsopossible to adopt a configuration in which a plurality of lightreceiving units 31 is formed into a block as a unit, one pixel signalgeneration unit 62 is provided for each pixel block, and the pixelsignal generation unit 62 is shared among the plurality of lightreceiving units 61 in each pixel block.

Example Applications of the Technology According to the PresentDisclosure

The technology according to the present disclosure may be applied tovarious products. In the description below, more specific exampleapplications are described. For example, the technology according to thepresent disclosure may be embodied as an imaging system or an objectrecognition system that is mounted on any type of mobile unit, such asan automobile, an electrical vehicle, a hybrid electrical vehicle, amotorcycle, a bicycle, a personal mobility device, an airplane, a drone,a vessel, a robot, a construction machine, or an agricultural machine (atractor).

Mobile Unit

FIG. 28 is a block diagram schematically showing an exampleconfiguration of a vehicle control system 7000 that is an example of amobile unit control system to which the technology according to thepresent disclosure can be applied. The vehicle control system 7000includes a plurality of electronic control units connected via acommunication network 7010. In the example shown in FIG. 28, the vehiclecontrol system 7000 includes a drive system control unit 7100, a bodysystem control unit 7200, a battery control unit 7300, an externalinformation detection unit 7400, an in-vehicle information detectionunit 7500, and an overall control unit 7600. The communication network7010 connecting the plurality of control units may be an in-vehiclecommunication network compliant with an appropriate standard, such as acontroller area network (CAN), a local interconnect network (LIN), alocal area network (LAN), or FlexRay (registered trademark), forexample.

Each of the control units includes: a microcomputer that performsarithmetic processing according to various programs; a storage unit thatstores the programs to be executed by the microcomputer, the parametersto be used for various calculations, or the like; and a drive circuitthat drives the current device to be subjected to various kinds ofcontrol. Each of the control units includes a communication interfacefor performing communication through wired communication or wirelesscommunication with an external device or a sensor or the like, as wellas a network interface for communicating with another control unit viathe communication network 7010. In FIG. 28, a microcomputer 7610, ageneral-purpose communication interface 7620, a dedicated communicationinterface 7630, a positioning unit 7640, a beacon reception unit 7650,an in-vehicle device interface 7660, a sound/image output unit 7670, anin-vehicle network interface 7680, and a storage unit 7690 are shown asthe functional components of the overall control unit 7600. Likewise,the other control units each include a microcomputer, a communicationinterface, a storage unit, and the like.

The drive system control unit 7100 controls operations of the devicesrelated to the drive system of the vehicle according to variousprograms. For example, the drive system control unit 7100 functions ascontrol devices such as a driving force generation device for generatinga driving force of the vehicle such as an internal combustion engine ora driving motor, a driving force transmission mechanism for transmittingthe driving force to the wheels, a steering mechanism for adjusting thesteering angle of the vehicle, and a braking device for generating abraking force of the vehicle. The drive system control unit 7100 mayalso have functions as a control device such as an antilock brake system(ABS) or an electronic stability control (ESC).

A vehicle state detector 7110 is connected to the drive system controlunit 7100. For example, the vehicle state detector 7110 includes atleast one of the following components: a gyroscope sensor that detectsan angular velocity of axial rotation motion of the vehicle body; anacceleration sensor that detects an acceleration of the vehicle; and asensor for detecting an operation amount of the gas pedal, an operationamount of the brake pedal, a steering angle of the steering wheel, anengine rotation speed, a wheel rotation speed, or the like. The drivesystem control unit 7100 performs arithmetic processing using a signalinput from the vehicle state detector 7110, and controls the internalcombustion engine, the driving motor, the electrical power steeringdevice, the brake device, or the like.

The body system control unit 7200 controls operations of the variousdevices mounted on the vehicle body according to various programs. Forexample, the body system control unit 7200 functions as a keyless entrysystem, a smart key system, a power window device, or a control devicefor various lamps such as a headlight, a backup light, a brake light, aturn signal light, or a fog light. In this case, the body system controlunit 7200 can receive radio waves transmitted from a portable devicethat substitutes for a key, or signals from various switches. The bodysystem control unit 7200 receives inputs of these radio waves orsignals, and controls the door lock device, the power window device, thelights, and the like of the vehicle.

The battery control unit 7300 controls a secondary battery 7310 that isa power supply source for the driving motor, according to variousprograms. For example, the battery control unit 7300 receivesinformation, such as a battery temperature, a battery output voltage, ora remaining capacity of the battery, from a battery device including thesecondary battery 7310. The battery control unit 7300 performsarithmetic processing using these signals, to control temperatureadjustment of the secondary battery 7310 or to control a cooling deviceor the like provided in the battery device.

The external information detection unit 7400 detects information outsidethe vehicle equipped with the vehicle control system 7000. For example,an imaging unit 7410 and/or an external information detector 7420 isconnected to the external information detection unit 7400. The imagingunit 7410 includes at least one of the following cameras: atime-of-flight (ToF) camera, a stereo camera, a monocular camera, aninfrared camera, or other cameras. The external information detector7420 includes an environment sensor for detecting the current weather ormeteorological phenomenon, and/or an ambient information detectionsensor for detecting another vehicle, an obstacle, a pedestrian, or thelike around the vehicle equipped with the vehicle control system 7000,for example.

The environment sensor may be formed with at least one of the followingsensors: a raindrop sensor that detects rain, a fog sensor that detectsa fog, a solar radiation sensor that detects a degree of solarradiation, or a snow sensor that detects a snowfall, for example. Theambient information detection sensor may be at least one of thefollowing devices: an ultrasonic sensor, a radar device, or a lightdetection and ranging, laser imaging detection and ranging (LIDAR)device. The imaging unit 7410 and the external information detector 7420may be provided as an independent device and an independent sensor,respectively, or may be provided as a device in which a plurality ofsensors or devices is integrated.

Here, FIG. 29 shows an example of installation positions of imagingunits 7410 and external information detectors 7420. Imaging units 7910,7912, 7914, 7916, and 7918 are provided at at least one of the followingpositions: the front end edge of a vehicle 7900, a side mirror, the rearbumper, a rear door, or an upper portion of the front windshield insidethe vehicle, for example. The imaging unit 7910 provided on the frontend edge and the imaging unit 7918 provided on the upper portion of thefront windshield inside the vehicle mainly capture images ahead of thevehicle 7900. The imaging units 7912 and 7914 provided on the sidemirrors mainly capture images on the sides of the vehicle 7900. Theimaging unit 7916 provided on the rear bumper or a rear door mainlycaptures images behind the vehicle 7900. The imaging unit 7918 providedon the upper portion of the front windshield inside the vehicle ismainly used for detection of a vehicle running in front of the vehicle,a pedestrian, an obstacle, a traffic signal, a traffic sign, a lane, orthe like.

Note that FIG. 29 shows an example of the imaging range of each of theimaging units 7910, 7912, 7914, and 7916. An imaging range a indicatesthe imaging range of the imaging unit 7910 provided on the front endedge, imaging ranges b and c indicate the imaging ranges of the imagingunits 7912 and 7914 provided on the respective side mirrors, and animaging range d indicates the imaging range of the imaging unit 7916provided on the rear bumper or a rear door. For example, image datacaptured by the imaging units 7910, 7912, 7914, and 7916 aresuperimposed on one another, so that an overhead image of the vehicle7900 viewed from above is obtained.

External information detectors 7920, 7922, 7924, 7926, 7928, and 7930provided on the front, the rear, the sides, and the corners of thevehicle 7900, and an upper portion of the front windshield inside thevehicle may be ultrasonic sensors or radar devices, for example. Theexternal information detectors 7920, 7926, and 7930 provided on thefront end edge of the vehicle 7900, the rear bumper, and the rear doors,and the upper portion of the front windshield inside the vehicle may beLIDAR devices, for example. These external information detectors 7920through 7930 are mainly used for detecting a vehicle running in front ofthe vehicle 7900, a pedestrian, an obstacle, or the like.

Referring back to FIG. 28, the explanation is continued. The externalinformation detection unit 7400 causes the imaging unit 7410 to capturean image of the outside of the vehicle, and receives the captured imagedata. The external information detection unit 7400 also receivesdetection information from the external information detector 7420connected thereto. In a case where the external information detector7420 is an ultrasonic sensor, a radar device, or a LIDAR device, theexternal information detection unit 7400 causes the external informationdetector 7420 to transmit ultrasonic waves, or electromagnetic waves, orthe like, and receive information about received reflected waves. On thebasis of the received information, the external information detectionunit 7400 may perform an object detection process for detecting aperson, a vehicle, an obstacle, a sign, characters on the road surface,or the like, or perform a distance detection process. On the basis ofthe received information, the external information detection unit 7400may also perform an environment recognition process for recognizing arainfall, a fog, a road surface condition, or the like. On the basis ofthe received information, the external information detection unit 7400may also calculate the distance to an object outside the vehicle.

Further, on the basis of the received image data, the externalinformation detection unit 7400 may perform an image recognition processfor recognizing a person, a vehicle, an obstacle, a sign, characters onthe road surface, or the like, or a distance detection process. Theexternal information detection unit 7400 may also perform processingsuch as distortion correction or positioning on the received image data,and combine the image data captured by different imaging units 7410, togenerate an overhead image or a panoramic image. The externalinformation detection unit 7400 may also perform a viewpoint conversionprocess, using image data captured by different imaging units 7410.

The in-vehicle information detection unit 7500 detects information aboutthe inside of the vehicle. For example, a driver state detector 7510that detects the state of the driver is connected to the in-vehicleinformation detection unit 7500. The driver state detector 7510 mayinclude a camera that captures images of the driver, a biometric sensorthat detects biological information about the driver, a microphone thatcollects sounds inside the vehicle, or the like. The biometric sensor isprovided on the seating surface or the steering wheel or the like, forexample, and detects biological information about a passenger sitting ona seat or the driver holding the steering wheel. On the basis of thedetection information input from the driver state detector 7510, thein-vehicle information detection unit 7500 may calculate the degree offatigue or the degree of concentration of the driver, or determinewhether the driver is dozing off. The in-vehicle information detectionunit 7500 may also perform a noise cancel process or the like on thecollected sound signals.

The overall control unit 7600 controls the entire operation in thevehicle control system 7000 according to various programs. An input unit7800 is connected to the overall control unit 7600. The input unit 7800is formed with a device on which a passenger can perform an inputoperation, such as a touch panel, buttons, a microphone, a switch, or alever, for example. The overall control unit 7600 may receive dataobtained by performing speech recognition on sound input through amicrophone. For example, the input unit 7800 may be a remote controldevice using infrared rays or some other radio waves, or an externalconnection device such as a portable telephone or a personal digitalassistant (PDA) compatible with operations on the vehicle control system7000. The input unit 7800 may be a camera, for example, and in thatcase, a passenger can input information by gesture. Alternatively, dataobtained by detecting movement of a wearable device worn by a passengermay be input. Further, the input unit 7800 may include an input controlcircuit or the like that generates an input signal on the basis ofinformation input by a passenger or the like using the above input unit7800, for example, and outputs the input signal to the overall controlunit 7600. By operating this input unit 7800, a passenger or the likeinputs various data to the vehicle control system 7000 or issues aprocessing operation instruction to the vehicle control system 7000.

The storage unit 7690 may include a read only memory (ROM) that storesvarious programs to be executed by the microcomputer, and a randomaccess memory (RAM) that stores various parameters, calculation results,sensor values, or the like. Also, the storage unit 7690 may be formedwith a magnetic storage device such as a hard disc drive (HDD), asemiconductor storage device, an optical storage device, amagneto-optical storage device, or the like.

The general-purpose communication interface 7620 is a generalcommunication interface that mediates communication with various devicesexisting in external environments 7750. The general-purposecommunication interface 7620 may implement a cellular communicationprotocol such as Global System of Mobile communications (GSM)(registered trademark), WiMAX, Long Term Evolution (LTE), orLTE-Advanced (LTE-A), or some other wireless communication protocol suchas wireless LAN (also called Wi-Fi (registered trademark)) or Bluetooth(registered trademark). The general-purpose communication interface 7620may be connected to a device (an application server or a control server,for example) existing in an external network (the Internet, a cloudnetwork, or a company-specific network, for example) via a base stationor an access point, for example. Alternatively, the general-purposecommunication interface 7620 may be connected to a terminal (a terminalof a driver, a pedestrian, or a shop, or a machine type communication(MTC) terminal, for example) existing in the vicinity of the vehicle,using the peer-to-peer (P2P) technology, for example.

The dedicated communication interface 7630 is a communication interfacethat supports a communication protocol formulated for use in a vehicle.The dedicated communication interface 7630 may implement a standardprotocol such as Wireless Access in Vehicle Environment (WAVE), which isa combination of IEEE802.11p as the lower layer and IEEE1609 as theupper layer, Dedicated Short Range Communications (DSRC), or a cellularcommunication protocol, for example. Typically, the dedicatedcommunication interface 7630 conducts V2X communication, which is aconcept including at least one of the following kinds of communication:vehicle-to-vehicle communication, vehicle-to-infrastructurecommunication, vehicle-to-home communication, and vehicle-to-pedestriancommunication.

The positioning unit 7640 receives a GNSS signal (a GPS signal from aglobal positioning system (GPS) satellite, for example) from a globalnavigation satellite system (GNSS) satellite, performs positioning, andgenerates location information including the latitude, the longitude,and the altitude of the vehicle, for example. Note that the positioningunit 7640 may identify the current location by exchanging signals with awireless access point, or may acquire the location information from aterminal having a positioning function, such as a portable telephone, aPHS, or a smartphone.

The beacon reception unit 7650 receives radio waves or electromagneticwaves transmitted from a wireless station or the like installed on aroad, for example, and acquires information about the current location,traffic congestion, closing of a road, a required time, or the like.Note that the functions of the beacon reception unit 7650 may beincluded in the dedicated communication interface 7630 described above.

The in-vehicle device interface 7660 is a communication interface thatmediates connection between the microcomputer 7610 and variousin-vehicle devices 7760 existing in the vehicle. The in-vehicle deviceinterface 7660 may establish a wireless connection, using a wirelesscommunication protocol such as wireless LAN, Bluetooth (registeredtrademark), Near Field Communication (NFC), or wireless USB (WUSB).Further, the in-vehicle device interface 7660 may establish a wiredconnection to a universal serial bus (USB), a high-definition multimediainterface (HDMI), a mobile high-definition link (MHL), or the like via aconnecting terminal (not shown) (and a cable, if necessary). Thein-vehicle devices 7760 may include a mobile device or a wearable deviceowned by a passenger, and/or an information device installed in orattached to the vehicle, for example. The in-vehicle devices 7760 mayalso include a navigation device that searches for a route to a desireddestination. The in-vehicle device interface 7660 exchanges controlsignals or data signals with these in-vehicle devices 7760.

The in-vehicle network interface 7680 is an interface that mediatescommunication between the microcomputer 7610 and the communicationnetwork 7010. The in-vehicle network interface 7680 transmits andreceives signals and the like, according to a predetermined protocolsupported by the communication network 7010.

The microcomputer 7610 of the overall control unit 7600 controls thevehicle control system 7000 according to various programs, on the basisof information acquired via at least one of the following components:the general-purpose communication interface 7620, the dedicatedcommunication interface 7630, the positioning unit 7640, the beaconreception unit 7650, the in-vehicle device interface 7660, and thein-vehicle network interface 7680. For example, on the basis of acquiredexternal and internal information, the microcomputer 7610 may calculatethe control target value of the driving force generation device, thesteering mechanism, or the braking device, and output a control commandto the drive system control unit 7100. For example, the microcomputer7610 may perform cooperative control to achieve the functions of anadvanced driver assistance system (ADAS), including vehicle collisionavoidance or impact mitigation, follow-up running based on the distancebetween vehicles, vehicle speed maintenance running, vehicle collisionwarning, vehicle lane deviation warning, or the like. The microcomputer7610 may also perform cooperative control to conduct automatic drivingor the like for autonomously running not depending on the operation ofthe driver, by controlling the driving force generation device, thesteering mechanism, the braking device, or the like on the basis ofacquired information about the surroundings of the vehicle.

The microcomputer 7610 may generate information about thethree-dimensional distance between the vehicle and an object such as anearby architectural structure or a person, and create local mapinformation including surroundings information about the currentlocation of the vehicle, on the basis of information acquired via atleast one of the following components: the general-purpose communicationinterface 7620, the dedicated communication interface 7630, thepositioning unit 7640, the beacon reception unit 7650, the in-vehicledevice interface 7660, and the in-vehicle network interface 7680. Themicrocomputer 7610 may also generate a warning signal by predictingdanger such as a collision of the vehicle, an approach of a pedestrianor the like, or entry to a closed road, on the basis of acquiredinformation. The warning signal may be a signal for generating an alarmsound or for turning on a warning lamp, for example.

The sound/image output unit 7670 transmits an audio output signal and/oran image output signal to an output device that is capable of visuallyor audibly notifying the passenger(s) of the vehicle or the outside ofthe vehicle of information. In the example shown in FIG. 28, an audiospeaker 7710, a display unit 7720, and an instrument panel 7730 areshown as output devices. The display unit 7720 may include an on-boarddisplay and/or a head-up display, for example. The display unit 7720 mayhave an augmented reality (AR) display function. An output device may besome device other than the above devices, such as a wearable device likea headphone set or an eyeglass-type display to be worn by a passenger, aprojector, or a lamp. In a case where the output device is a displaydevice, the display device visually displays results obtained throughvarious processes performed by the microcomputer 7610, or informationreceived from other control units, in various forms such as text, animage, a table, or a graph. Further, in a case where the output deviceis a sound output device, the sound output device converts an audiosignal formed with reproduced sound data, acoustic data, or the likeinto an analog signal, and audibly outputs the analog signal.

Note that, in the example shown in FIG. 28, at least two control unitsconnected via the communication network 7010 may be integrated into onecontrol unit. Alternatively, each control unit may be formed with aplurality of control units. Further, the vehicle control system 7000 mayinclude another control unit that is not shown in the drawing. Also, inthe above description, some or all of the functions of one of thecontrol units may be provided by some other control unit. That is, aslong as information is transmitted and received via the communicationnetwork 7010, predetermined arithmetic processing may be performed byany control unit. Likewise, a sensor or a device connected to anycontrol unit may be connected to another control unit, and a pluralityof control units may transmit and receive detection information to andfrom one another via the communication network 7010.

An example of a vehicle control system to which the technology accordingto the present disclosure can be applied has been described above. Thetechnology according to the present disclosure can be applied to theimaging units 7910, 7912, 7914, 7916, and 7918, and the like among thecomponents described above, for example. Specifically, an imaging systemof the present disclosure can be applied to these imaging units. Animaging system of the present disclosure can accurately recognize anobject in an event without being affected by bad weather such as rain orsnow, and thus, contribute to realization of safe vehicle traveling.

Configurations Embodying the Present Disclosure

Note that the present disclosure may also be embodied in theconfigurations described below.

<A. Imaging System>

[A-1] An imaging system including:

an event detection device that detects an event; and

a controller that controls the event detection device,

in which the controller controls detection sensitivity of eventdetection being performed by the event detection device, on the basis ofexternal information.

[A-2] The imaging system according to [A-1], in which

the event detection device includes an event detection unit thatdetects, as an event, that a change in the luminance of a pixel thatphotoelectrically converts incident light exceeds a detection threshold.

[A-3] The imaging system according to [A-2],

which is mounted on a mobile unit and is used.

[A-4] The imaging system according to [A-3], in which

the controller controls the detection threshold for the event detectionunit, on the basis of the external information.

[A-5] The imaging system according to [A-4], in which,

when the external information is information indicating bad weather, thecontroller performs control to raise the detection threshold for theevent detection unit.

[A-6] The imaging system according to [A-5], in which,

when receiving the external information indicating that weather hasrecovered after raising the detection threshold for the event detectionunit, the controller performs control to lower the detection thresholdfor the event detection unit or return the detection threshold for theevent detection unit to an initial setting value.

[A-7] The imaging system according to any one of [A-4] to [A-6], inwhich

the event detection unit includes a current-voltage conversion unit thatconverts a photocurrent of a pixel into a voltage corresponding to thephotocurrent,

the current-voltage conversion unit is capable of switching between acircuit configuration in which transistors are cascade-connected and acircuit configuration in which the transistors are notcascade-connected, and

the controller controls the detection threshold for the event detectionunit, by switching the circuit configuration of the current-voltageconversion unit, on the basis of the external information.

[A-8] The imaging system according to any one of [A-4] to [A-6], inwhich

the event detection unit includes a subtraction unit that includes afirst capacitive element and a second capacitive element, and calculatesa difference signal of a voltage at different timings, the voltagecorresponding to a photocurrent of a pixel, and

the controller controls the detection threshold for the event detectionunit, by changing a capacitance ratio between the first capacitiveelement and the second capacitive element of the subtraction unit, onthe basis of the external information.

[A-9] The imaging system according to [A-8], in which

the event detection unit includes a quantization unit that quantizes adifference signal from the subtraction unit into a digital signal bycomparing the difference signal with a threshold voltage, and

the controller controls the detection threshold for the event detectionunit, by adjusting the threshold voltage of the quantization unit on thebasis of the external information.

[A-10] The imaging system according to any one of [A-1] to [A-9], inwhich

the controller controls the detection sensitivity of the eventdetection, on the basis of the external information and the number ofevents detected by the event detection device.

<B. Object Recognition System>

[B-1] An object recognition system including:

an event detection device that detects an event;

a controller that controls detection sensitivity of event detectionbeing performed by the event detection device, on the basis of externalinformation; and

a recognition processing unit that performs object recognition within anangle of view of the event detection device, on the basis of an eventsignal output from the event detection device.

[B-2] The imaging system according to [B-1], in which

the controller performs control to raise a detection threshold for anevent detection unit to a value within a range in which an object withinthe angle of view of the event detection device can be recognized.

[B-3] The object recognition system according to [B-1], in which

the recognition processing unit recognizes an object when an area thatcan be detected as the object within the angle of view is equal to orlarger than a predetermined threshold.

[B-4] The object recognition system according to [B-1], furtherincluding

a synchronous imaging device that performs imaging at a fixed framerate.

[B-5] The object recognition system according to [B-4], in which,

when the object cannot be detected by the event detection device, therecognition processing unit detects the object using information fromthe imaging device, and recognizes the object when an area that can bedetected as the object within the angle of view is equal to or largerthan a predetermined threshold.

[B-6] The object recognition system according to [B-4] or [B-5], inwhich

the recognition processing unit determines that recognition has beensuccessfully performed when a region of the object with respect to theangle of view is equal to or larger than a certain proportion, anddetermines that recognition has failed when the region is smaller thanthe certain proportion.

[B-7] The object recognition system according to any one of [B-4] to[B-6], in which,

when the external information is information indicating bad weather, andthe number of events detected by the event detection device is equal toor greater than a predetermined threshold, the controller performscontrol to raise the detection threshold for the event detection unit.

[B-8] The object recognition system according to [B-7], in which,

after the controller raises the detection threshold for the eventdetection unit, when a result of recognition performed by therecognition processing unit indicates a success, and the controllerreceives the external information indicating that weather has recovered,the controller performs control to lower the detection threshold for theevent detection unit or return the detection threshold for the eventdetection unit to an initial setting value.

[B-9] The object recognition system according to any one of [B-4] to[B-6], in which,

when the external information is information indicating bad weather, anda result of recognition performed by the recognition processing unitindicates a failure, the controller performs control to raise thedetection threshold for the event detection unit.

[B-10] The object recognition system according to [B-9], in which,

after the controller raises the detection threshold for the eventdetection unit, when a result of recognition performed by therecognition processing unit indicates a success, and the controllerreceives the external information indicating that weather has recovered,the controller performs control to lower the detection threshold for theevent detection unit or return the detection threshold for the eventdetection unit to an initial setting value.

REFERENCE SIGNS LIST

-   1A Imaging system according to the first embodiment-   1B Imaging system according to the second embodiment-   2A Object recognition system according to the first embodiment-   2B Object recognition system according to the second embodiment-   10 Event detection device-   11 Pixel-   12 Pixel array unit-   13 Drive unit-   14 Arbiter unit (arbitration unit)-   15 Column processing unit-   16 Signal processing unit-   Imaging device-   21 Pixel-   22 Pixel array unit-   23 Row selection unit-   24 Constant-current supply unit-   25 Analog-digital conversion unit-   26 Horizontal transfer scanning unit-   27 Signal processing unit-   28 Timing control unit-   30 Controller-   40 Data processing unit-   50 Image recording unit-   60 Recognition processing unit-   61 Light receiving unit-   62 Pixel signal generation unit-   63 Event detection unit

1. An imaging system comprising: an event detection device that detectsan event; and a controller that controls the event detection device,wherein the controller controls detection sensitivity of event detectionbeing performed by the event detection device, on a basis of externalinformation.
 2. The imaging system according to claim 1, wherein theevent detection device includes an event detection unit that detects, asan event, that a change in the luminance of a pixel thatphotoelectrically converts incident light exceeds a detection threshold.3. The imaging system according to claim 2, which is mounted on a mobileunit and is used.
 4. The imaging system according to claim 3, whereinthe controller controls the detection threshold for the event detectionunit, on a basis of the external information.
 5. The imaging systemaccording to claim 4, wherein, when the external information isinformation indicating bad weather, the controller performs control toraise the detection threshold for the event detection unit.
 6. Theimaging system according to claim 5, wherein, when receiving theexternal information indicating that weather has recovered after raisingthe detection threshold for the event detection unit, the controllerperforms control to lower the detection threshold for the eventdetection unit or return the detection threshold for the event detectionunit to an initial setting value.
 7. The imaging system according toclaim 4, wherein the event detection unit includes a current-voltageconversion unit that converts a photocurrent of a pixel into a voltagecorresponding to the photocurrent, the current-voltage conversion unitis capable of switching between a circuit configuration in whichtransistors are cascade-connected and a circuit configuration in whichthe transistors are not cascade-connected, and the controller controlsthe detection threshold for the event detection unit, by switching acircuit configuration of the current-voltage conversion unit, on a basisof the external information.
 8. The imaging system according to claim 4,wherein the event detection unit includes a subtraction unit thatincludes a first capacitive element and a second capacitive element, andcalculates a difference signal of a voltage at different timings, thevoltage corresponding to a photocurrent of a pixel, and the controllercontrols the detection threshold for the event detection unit, bychanging a capacitance ratio between the first capacitive element andthe second capacitive element of the subtraction unit, on a basis of theexternal information.
 9. The imaging system according to claim 8,wherein the event detection unit includes a quantization unit thatquantizes a difference signal from the subtraction unit into a digitalsignal by comparing the difference signal with a threshold voltage, andthe controller controls the detection threshold for the event detectionunit, by adjusting the threshold voltage of the quantization unit on abasis of the external information.
 10. The imaging system according toclaim 1, wherein the controller controls the detection sensitivity ofthe event detection, on a basis of the external information and thenumber of events detected by the event detection device.
 11. A methodfor controlling an imaging system that includes an event detectiondevice that detects an event, the method comprising controllingdetection sensitivity of event detection being performed by the eventdetection device, on a basis of external information.
 12. An objectrecognition system comprising: an event detection device that detects anevent; a controller that controls detection sensitivity of eventdetection being performed by the event detection device, on a basis ofexternal information; and a recognition processing unit that performsobject recognition within an angle of view of the event detectiondevice, on a basis of an event signal output from the event detectiondevice.
 13. The imaging system according to claim 12, wherein thecontroller performs control to raise a detection threshold for an eventdetection unit to a value within a range in which an object within theangle of view of the event detection device can be recognized.
 14. Theobject recognition system according to claim 12, wherein the recognitionprocessing unit recognizes an object when an area that can be detectedas the object within the angle of view is equal to or larger than apredetermined threshold.
 15. The object recognition system according toclaim 12, further comprising a synchronous imaging device that performsimaging at a fixed frame rate.
 16. The object recognition systemaccording to claim 15, wherein, when the object cannot be detected bythe event detection device, the recognition processing unit detects theobject using information from the imaging device, and recognizes theobject when an area that can be detected as the object within the angleof view is equal to or larger than a predetermined threshold.
 17. Theobject recognition system according to claim 15, wherein the recognitionprocessing unit determines that recognition has been successfullyperformed when a region of the object with respect to the angle of viewis equal to or larger than a certain proportion, and determines thatrecognition has failed when the region is smaller than the certainproportion.
 18. The object recognition system according to claim 15,wherein, when the external information is information indicating badweather, and the number of events detected by the event detection deviceis equal to or greater than a predetermined threshold, the controllerperforms control to raise the detection threshold for the eventdetection unit.
 19. The object recognition system according to claim 18,wherein, after the controller raises the detection threshold for theevent detection unit, when a result of recognition performed by therecognition processing unit indicates a success, and the controllerreceives the external information indicating that weather has recovered,the controller performs control to lower the detection threshold for theevent detection unit or return the detection threshold for the eventdetection unit to an initial setting value.
 20. The object recognitionsystem according to claim 15, wherein, when the external information isinformation indicating bad weather, and a result of recognitionperformed by the recognition processing unit indicates a failure, thecontroller performs control to raise the detection threshold for theevent detection unit.
 21. The object recognition system according toclaim 20, wherein, after the controller raises the detection thresholdfor the event detection unit, when a result of recognition performed bythe recognition processing unit indicates a success, and the controllerreceives the external information indicating that weather has recovered,the controller performs control to lower the detection threshold for theevent detection unit or return the detection threshold for the eventdetection unit to an initial setting value.